[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:15:55 dist-test-slave-s1hw krb5kdc[1050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:15:55 dist-test-slave-s1hw krb5kdc[1050](info): set up 2 sockets
Sep 24 23:15:55 dist-test-slave-s1hw krb5kdc[1050](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 23:15:56.424300 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:15:56 dist-test-slave-s1hw krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:15:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:15:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:15:56.542501 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:33889
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:33889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:15:56.622136 1063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:15:56.622359 1063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:15:56.622390 1063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:15:56.623605 1063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:15:56.623651 1063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:15:56.623668 1063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:15:56.623684 1063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:15:56.623698 1063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:15:56.625205 1063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:33889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:33889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:15:56.625459 1063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:15:56.625694 1063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:15:56.630716 1069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:15:56.630810 1068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:15:56.630868 1063 server_base.cc:1034] running on GCE node
W20260924 23:15:56.630671 1071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:15:56.631212 1063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:15:56.631542 1063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:15:56.632684 1063 hybrid_clock.cc:648] HybridClock initialized: now 1790291756632661 us; error 43 us; skew 500 ppm
Sep 24 23:15:56 dist-test-slave-s1hw krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291756, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:15:56.635126 1063 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:15:56.635620 1063 webserver.cc:460] Webserver started at http://127.1.4.254:32911/ using document root <none> and password file <none>
I20260924 23:15:56.635809 1063 fs_manager.cc:359] Metadata directory not provided
I20260924 23:15:56.635848 1063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:15:56.635948 1063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:15:56.637008 1063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "dc967933e00c4adebaf6c821381e763a"
format_stamp: "Formatted at 2026-09-24 23:15:56 on dist-test-slave-s1hw"
server_key: "2e56d484981519e99674e11115130ef0"
server_key_iv: "4ae181d9d83b21d6652714cbe4ff9e7e"
server_key_version: "encryptionkey@0"
I20260924 23:15:56.637316 1063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "dc967933e00c4adebaf6c821381e763a"
format_stamp: "Formatted at 2026-09-24 23:15:56 on dist-test-slave-s1hw"
server_key: "2e56d484981519e99674e11115130ef0"
server_key_iv: "4ae181d9d83b21d6652714cbe4ff9e7e"
server_key_version: "encryptionkey@0"
I20260924 23:15:56.638592 1063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 23:15:56.639246 1078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:56.639423 1063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:56.639528 1063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "dc967933e00c4adebaf6c821381e763a"
format_stamp: "Formatted at 2026-09-24 23:15:56 on dist-test-slave-s1hw"
server_key: "2e56d484981519e99674e11115130ef0"
server_key_iv: "4ae181d9d83b21d6652714cbe4ff9e7e"
server_key_version: "encryptionkey@0"
I20260924 23:15:56.639585 1063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:15:56.656661 1063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:15:56.658833 1063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:15:56.658951 1063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:15:56.662919 1063 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:33889
I20260924 23:15:56.662953 1130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:33889 every 8 connection(s)
I20260924 23:15:56.663275 1063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:15:56.664158 1131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:15:56.666441 1131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a: Bootstrap starting.
I20260924 23:15:56.667001 1131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a: Neither blocks nor log segments found. Creating new log.
I20260924 23:15:56.667258 1131 log.cc:824] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:15:56.667554 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 1063
I20260924 23:15:56.667670 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:15:56.667950 1131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a: No bootstrap required, opened a new log
I20260924 23:15:56.668414 1043 external_mini_cluster.cc:1428] Setting key 047cfeaeb23f33c3bc5ecb3b3f3924da
I20260924 23:15:56.669711 1131 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc967933e00c4adebaf6c821381e763a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33889 } }
I20260924 23:15:56.669978 1131 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:15:56.670013 1131 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc967933e00c4adebaf6c821381e763a, State: Initialized, Role: FOLLOWER
I20260924 23:15:56.670138 1131 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc967933e00c4adebaf6c821381e763a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33889 } }
I20260924 23:15:56.670222 1131 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:15:56.670287 1131 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:15:56.670338 1131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:15:56.671480 1131 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc967933e00c4adebaf6c821381e763a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33889 } }
I20260924 23:15:56.671610 1131 leader_election.cc:302] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc967933e00c4adebaf6c821381e763a; no voters:
I20260924 23:15:56.671784 1131 leader_election.cc:288] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:15:56.671857 1136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:15:56.672056 1131 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:15:56.672099 1136 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [term 1 LEADER]: Becoming Leader. State: Replica: dc967933e00c4adebaf6c821381e763a, State: Running, Role: LEADER
I20260924 23:15:56.672266 1136 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc967933e00c4adebaf6c821381e763a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33889 } }
I20260924 23:15:56.673295 1138 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc967933e00c4adebaf6c821381e763a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc967933e00c4adebaf6c821381e763a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33889 } } }
W20260924 23:15:56.673813 1150 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:15:56.673856 1150 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:15:56.673404 1137 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc967933e00c4adebaf6c821381e763a. Latest consensus state: current_term: 1 leader_uuid: "dc967933e00c4adebaf6c821381e763a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc967933e00c4adebaf6c821381e763a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33889 } } }
I20260924 23:15:56.673843 1138 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [sys.catalog]: This master's current role is: LEADER
I20260924 23:15:56.673919 1137 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a [sys.catalog]: This master's current role is: LEADER
I20260924 23:15:56.674093 1153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:15:56.674235 1153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:15:56.675772 1153 catalog_manager.cc:1347] Generated new cluster ID: 15cdc5a2c7804545ace1ed5651b5dfa3
I20260924 23:15:56.675823 1153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:15:56 dist-test-slave-s1hw krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:15:56.681948 1153 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:15:56.682649 1153 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:15:56.686168 1153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc967933e00c4adebaf6c821381e763a: Generated new TSK 0
I20260924 23:15:56.686337 1153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:15:56.686941 1134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:56.669282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49902 (local address 127.1.4.254:33889)
0924 23:15:56.669504 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:56.669514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:56.670236 (+ 722us) server_negotiation.cc:407] Connection header received
0924 23:15:56.672602 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:56.672610 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:56.672676 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:56.672804 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:15:56.674790 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:56.676119 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:56.677621 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:56.678260 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:56.678356 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:56.682643 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:15:56.682648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:15:56.682648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:15:56.682661 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:15:56.684791 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:56.685325 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:56.685329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:56.685329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:56.685440 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:56.685713 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:56.685714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:56.685714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:56.686000 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:15:56.686135 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:56.686558 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 23:15:56.686714 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:15:56.711941 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33889
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:15:56.795259 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:15:56.795532 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:15:56.795559 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:15:56.796787 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:15:56.796832 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:15:56.796860 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:15:56.798322 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33889
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:15:56.798560 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:15:56.798789 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:15:56.803751 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:15:56.803861 1157 server_base.cc:1034] running on GCE node
W20260924 23:15:56.803790 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:15:56.803982 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:15:56.804178 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:15:56.804414 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:15:56.805546 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790291756805534 us; error 27 us; skew 500 ppm
Sep 24 23:15:56 dist-test-slave-s1hw krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291756, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:15:56.807991 1157 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:15:56.808413 1157 webserver.cc:460] Webserver started at http://127.1.4.193:43259/ using document root <none> and password file <none>
I20260924 23:15:56.808612 1157 fs_manager.cc:359] Metadata directory not provided
I20260924 23:15:56.808655 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:15:56.808775 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:15:56.809800 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "6387c2442f8f4b20801d1165a66a8261"
format_stamp: "Formatted at 2026-09-24 23:15:56 on dist-test-slave-s1hw"
server_key: "ef30a414f3a8eaeec08733b9e4ad084b"
server_key_iv: "f89ecb7c81581ff023eaf03e03b1f596"
server_key_version: "encryptionkey@0"
I20260924 23:15:56.810122 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "6387c2442f8f4b20801d1165a66a8261"
format_stamp: "Formatted at 2026-09-24 23:15:56 on dist-test-slave-s1hw"
server_key: "ef30a414f3a8eaeec08733b9e4ad084b"
server_key_iv: "f89ecb7c81581ff023eaf03e03b1f596"
server_key_version: "encryptionkey@0"
I20260924 23:15:56.811473 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:15:56.812209 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:56.812376 1157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:15:56.812448 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "6387c2442f8f4b20801d1165a66a8261"
format_stamp: "Formatted at 2026-09-24 23:15:56 on dist-test-slave-s1hw"
server_key: "ef30a414f3a8eaeec08733b9e4ad084b"
server_key_iv: "f89ecb7c81581ff023eaf03e03b1f596"
server_key_version: "encryptionkey@0"
I20260924 23:15:56.812532 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:15:56.830133 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:15:56.832366 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:15:56.832496 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:15:56.832801 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:15:56.833132 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:15:56.833169 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:56.833191 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:15:56.833212 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:56.839717 1157 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:45749
I20260924 23:15:56.839782 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:45749 every 8 connection(s)
I20260924 23:15:56.840078 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:15:56 dist-test-slave-s1hw krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291756, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:15:56.846966 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 1157
I20260924 23:15:56.847056 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:15:56.847191 1043 external_mini_cluster.cc:1428] Setting key c51a8e3ed982c0c4eaad1993ce872261
I20260924 23:15:56.849895 1134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:56.840789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:54779 (local address 127.1.4.254:33889)
0924 23:15:56.840874 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:56.840876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:56.841032 (+ 156us) server_negotiation.cc:407] Connection header received
0924 23:15:56.841528 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:56.841530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:56.841541 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:56.841600 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:15:56.842595 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:56.843242 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:56.844343 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:56.844544 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:56.844609 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:56.846677 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:15:56.846679 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:15:56.846679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:15:56.846686 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:15:56.848322 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:56.848800 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:56.848800 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:56.848801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:56.848851 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:56.849138 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:56.849139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:56.849139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:56.849275 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:15:56.849346 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:56.849698 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 23:15:56.849791 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 23:15:56.849959 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:56.840760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33889 (local address 127.1.4.193:54779)
0924 23:15:56.840911 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 23:15:56.840947 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 23:15:56.841408 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:15:56.841655 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:15:56.841658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:15:56.842274 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:15:56.842477 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:56.842482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:56.843352 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:56.843353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:56.844234 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:56.844236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:56.844616 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:56.844617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:56.844715 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:56.845179 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:15:56.845187 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:15:56.846590 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:15:56.848418 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:15:56.848435 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:15:56.848437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:15:56.848728 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:15:56.848903 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:15:56.848904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:15:56.848905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:15:56.849044 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:15:56.849340 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:15:56.849344 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:15:56.849557 (+ 213us) client_negotiation.cc:769] Sending connection context
0924 23:15:56.849660 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 23:15:56.849789 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260924 23:15:56.850476 1286 heartbeater.cc:347] Connected to a master server at 127.1.4.254:33889
I20260924 23:15:56.850569 1286 heartbeater.cc:464] Registering TS with master...
I20260924 23:15:56.850764 1286 heartbeater.cc:511] Master 127.1.4.254:33889 requested a full tablet report, sending...
I20260924 23:15:56.851262 1095 ts_manager.cc:194] Registered new tserver with Master: 6387c2442f8f4b20801d1165a66a8261 (127.1.4.193:45749)
I20260924 23:15:56.851945 1095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:54779
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:15:56.872659 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33889
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:15:56.955864 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:15:56.956091 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:15:56.956125 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:15:56.957300 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:15:56.957345 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:15:56.957382 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:15:56.958855 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33889
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:15:56.959120 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:15:56.959342 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:15:56.964327 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:15:56.964324 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:15:56.964358 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:15:56.964449 1291 server_base.cc:1034] running on GCE node
I20260924 23:15:56.964717 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:15:56.964944 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:15:56.966107 1291 hybrid_clock.cc:648] HybridClock initialized: now 1790291756966087 us; error 30 us; skew 500 ppm
Sep 24 23:15:56 dist-test-slave-s1hw krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291756, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:15:56.968667 1291 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:15:56.969090 1291 webserver.cc:460] Webserver started at http://127.1.4.194:46007/ using document root <none> and password file <none>
I20260924 23:15:56.969298 1291 fs_manager.cc:359] Metadata directory not provided
I20260924 23:15:56.969347 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:15:56.969468 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:15:56.970502 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "18789b7510c0432e9eacccc30b143e69"
format_stamp: "Formatted at 2026-09-24 23:15:56 on dist-test-slave-s1hw"
server_key: "62d0a2261cbc71674df9b3958059bfdc"
server_key_iv: "eb3940fdc5e7d503c3d882acacad6445"
server_key_version: "encryptionkey@0"
I20260924 23:15:56.970865 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "18789b7510c0432e9eacccc30b143e69"
format_stamp: "Formatted at 2026-09-24 23:15:56 on dist-test-slave-s1hw"
server_key: "62d0a2261cbc71674df9b3958059bfdc"
server_key_iv: "eb3940fdc5e7d503c3d882acacad6445"
server_key_version: "encryptionkey@0"
I20260924 23:15:56.972290 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:15:56.973137 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:56.973311 1291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:15:56.973395 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "18789b7510c0432e9eacccc30b143e69"
format_stamp: "Formatted at 2026-09-24 23:15:56 on dist-test-slave-s1hw"
server_key: "62d0a2261cbc71674df9b3958059bfdc"
server_key_iv: "eb3940fdc5e7d503c3d882acacad6445"
server_key_version: "encryptionkey@0"
I20260924 23:15:56.973491 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:15:56.988057 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:15:56.990097 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:15:56.990214 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:15:56.990407 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:15:56.990711 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:15:56.990742 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:56.990764 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:15:56.990775 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:56.996682 1291 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:34399
I20260924 23:15:56.996753 1419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:34399 every 8 connection(s)
I20260924 23:15:56.997088 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:15:56.997457 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 1291
I20260924 23:15:56.997587 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:15:56.997761 1043 external_mini_cluster.cc:1428] Setting key 48fa880c36965b4d67d399bfaa7395f6
Sep 24 23:15:56 dist-test-slave-s1hw krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291756, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
I20260924 23:15:57.006301 1134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:56.997840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:44625 (local address 127.1.4.254:33889)
0924 23:15:56.997899 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:56.997902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:56.998151 (+ 249us) server_negotiation.cc:407] Connection header received
0924 23:15:56.998732 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:56.998733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:56.998744 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:56.998800 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:15:56.999829 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:57.000330 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:57.001324 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:57.001528 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:57.001583 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:57.003648 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:15:57.003649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:15:57.003649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:15:57.003657 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:15:57.005043 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:57.005411 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:57.005412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:57.005412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:57.005466 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:57.005687 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:57.005688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:57.005688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:57.005800 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:15:57.005853 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:57.006130 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 23:15:57.006207 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:15:57.006491 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:56.997842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33889 (local address 127.1.4.194:44625)
0924 23:15:56.998035 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 23:15:56.998073 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 23:15:56.998634 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:15:56.998865 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:15:56.998869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:15:56.999529 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:15:56.999744 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:56.999749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:57.000410 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:57.000411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:57.001246 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:57.001248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:57.001595 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:57.001596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:57.001721 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:57.002190 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:15:57.002198 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:15:57.003558 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:15:57.005110 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:15:57.005118 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:15:57.005120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:15:57.005345 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:15:57.005520 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:15:57.005521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:15:57.005521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:15:57.005632 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:15:57.005852 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:15:57.005858 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:15:57.006003 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 23:15:57.006103 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 23:15:57.006271 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260924 23:15:57.006946 1420 heartbeater.cc:347] Connected to a master server at 127.1.4.254:33889
I20260924 23:15:57.007048 1420 heartbeater.cc:464] Registering TS with master...
I20260924 23:15:57.007223 1420 heartbeater.cc:511] Master 127.1.4.254:33889 requested a full tablet report, sending...
I20260924 23:15:57.007668 1095 ts_manager.cc:194] Registered new tserver with Master: 18789b7510c0432e9eacccc30b143e69 (127.1.4.194:34399)
I20260924 23:15:57.008100 1095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:44625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:15:57.022584 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33889
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:15:57.105176 1425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:15:57.105397 1425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:15:57.105422 1425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:15:57.106621 1425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:15:57.106668 1425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:15:57.106696 1425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:15:57.108249 1425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33889
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:15:57.108464 1425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:15:57.108651 1425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:15:57.113536 1431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:15:57.113633 1425 server_base.cc:1034] running on GCE node
W20260924 23:15:57.113600 1433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:15:57.113596 1430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:15:57.114037 1425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:15:57.114311 1425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:15:57.115417 1425 hybrid_clock.cc:648] HybridClock initialized: now 1790291757115398 us; error 31 us; skew 500 ppm
Sep 24 23:15:57 dist-test-slave-s1hw krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291757, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:15:57.117846 1425 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:15:57.118389 1425 webserver.cc:460] Webserver started at http://127.1.4.195:38933/ using document root <none> and password file <none>
I20260924 23:15:57.118592 1425 fs_manager.cc:359] Metadata directory not provided
I20260924 23:15:57.118637 1425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:15:57.118767 1425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:15:57.119876 1425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "279fd94a59b9452d9fa038a4972b513e"
format_stamp: "Formatted at 2026-09-24 23:15:57 on dist-test-slave-s1hw"
server_key: "17c790a6ba028bfbdbe1c6a8f39cfbc4"
server_key_iv: "02a6433046bb522f3adf4ae1d49c83fb"
server_key_version: "encryptionkey@0"
I20260924 23:15:57.120213 1425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "279fd94a59b9452d9fa038a4972b513e"
format_stamp: "Formatted at 2026-09-24 23:15:57 on dist-test-slave-s1hw"
server_key: "17c790a6ba028bfbdbe1c6a8f39cfbc4"
server_key_iv: "02a6433046bb522f3adf4ae1d49c83fb"
server_key_version: "encryptionkey@0"
I20260924 23:15:57.121536 1425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:15:57.122300 1440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:57.122457 1425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:15:57.122538 1425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "279fd94a59b9452d9fa038a4972b513e"
format_stamp: "Formatted at 2026-09-24 23:15:57 on dist-test-slave-s1hw"
server_key: "17c790a6ba028bfbdbe1c6a8f39cfbc4"
server_key_iv: "02a6433046bb522f3adf4ae1d49c83fb"
server_key_version: "encryptionkey@0"
I20260924 23:15:57.122596 1425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:15:57.138900 1425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:15:57.141101 1425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:15:57.141239 1425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:15:57.141462 1425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:15:57.141755 1425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:15:57.141788 1425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:57.141819 1425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:15:57.141835 1425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:57.147688 1425 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:40805
I20260924 23:15:57.147740 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:40805 every 8 connection(s)
I20260924 23:15:57.148049 1425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
Sep 24 23:15:57 dist-test-slave-s1hw krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291757, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:15:57.156950 1134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:57.148685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:41061 (local address 127.1.4.254:33889)
0924 23:15:57.148739 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:57.148741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:57.148925 (+ 184us) server_negotiation.cc:407] Connection header received
0924 23:15:57.149487 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:57.149488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:57.149500 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:57.149583 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:15:57.150501 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:57.151005 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:57.152021 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:57.152212 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:57.152267 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:57.154383 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:15:57.154384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:15:57.154385 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:15:57.154392 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:15:57.155713 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:57.156098 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:57.156099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:57.156100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:57.156151 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:57.156359 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:57.156360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:57.156360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:57.156466 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:15:57.156527 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:57.156781 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 23:15:57.156858 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:15:57.157136 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:57.148654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33889 (local address 127.1.4.195:41061)
0924 23:15:57.148811 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 23:15:57.148846 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:15:57.149387 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:15:57.149658 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:15:57.149662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:15:57.150218 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:15:57.150409 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:57.150414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:57.151090 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:57.151091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:57.151942 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:57.151944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:57.152277 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:57.152278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:57.152391 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:57.152887 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:15:57.152895 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:15:57.154292 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:15:57.155793 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:15:57.155801 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:15:57.155803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:15:57.156031 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:15:57.156203 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:15:57.156204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:15:57.156204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:15:57.156304 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:15:57.156518 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:15:57.156524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:15:57.156662 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 23:15:57.156752 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 23:15:57.156910 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260924 23:15:57.157207 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 1425
I20260924 23:15:57.157285 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:15:57.157441 1043 external_mini_cluster.cc:1428] Setting key 3dedba8c9028a1d1f1cbec82d9b6d1ee
I20260924 23:15:57.157722 1554 heartbeater.cc:347] Connected to a master server at 127.1.4.254:33889
I20260924 23:15:57.157831 1554 heartbeater.cc:464] Registering TS with master...
I20260924 23:15:57.158109 1554 heartbeater.cc:511] Master 127.1.4.254:33889 requested a full tablet report, sending...
I20260924 23:15:57.158499 1095 ts_manager.cc:194] Registered new tserver with Master: 279fd94a59b9452d9fa038a4972b513e (127.1.4.195:40805)
I20260924 23:15:57.158872 1095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:41061
I20260924 23:15:57.169544 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 23:15:57 dist-test-slave-s1hw krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 23:15:57 dist-test-slave-s1hw krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.4.193@KRBTEST.COM
I20260924 23:15:57.192184 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:57.184224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.1.4.193:45749)
0924 23:15:57.184358 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:57.184363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:57.184373 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:15:57.184399 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:57.184401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:57.184414 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:57.184489 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:15:57.185006 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:57.185764 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:57.186650 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:57.187003 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:57.187064 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:57.189297 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:15:57.189307 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:15:57.189308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:15:57.189316 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:15:57.191030 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:57.191402 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:57.191406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:57.191406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:57.191478 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:57.191649 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:57.191649 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:57.191650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:57.191799 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:15:57.191938 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:57.191993 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 23:15:57.192073 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
W20260924 23:15:57.192565 1200 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58230
Sep 24 23:15:57 dist-test-slave-s1hw krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 23:15:57 dist-test-slave-s1hw krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.193@KRBTEST.COM
I20260924 23:15:57.214505 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:57.207433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.1.4.193:45749)
0924 23:15:57.207490 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:57.207492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:57.207565 (+ 73us) server_negotiation.cc:407] Connection header received
0924 23:15:57.207638 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:57.207639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:57.207653 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:57.207701 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:15:57.208320 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:57.208811 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:57.209738 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:57.209954 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:57.210010 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:57.212170 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:15:57.212172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:15:57.212172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:15:57.212180 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:15:57.213394 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:57.213734 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:57.213736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:57.213736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:57.213790 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:57.214034 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:57.214035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:57.214035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:57.214175 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:15:57.214281 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:57.214334 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 23:15:57.214418 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 23:15:57.215027 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 1157
I20260924 23:15:57.219244 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 1291
I20260924 23:15:57.223807 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 1425
I20260924 23:15:57.227960 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 1063
2026-09-24T23:15:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1777 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:15:57 dist-test-slave-s1hw krb5kdc[1568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:15:57 dist-test-slave-s1hw krb5kdc[1568](info): set up 2 sockets
Sep 24 23:15:57 dist-test-slave-s1hw krb5kdc[1568](info): commencing operation
krb5kdc: starting...
W20260924 23:15:58.186797 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:15:58 dist-test-slave-s1hw krb5kdc[1568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:15:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:15:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:15:58.302440 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38899
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:36269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:38899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:15:58.381587 1581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:15:58.381812 1581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:15:58.381834 1581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:15:58.383028 1581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:15:58.383078 1581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:15:58.383091 1581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:15:58.383103 1581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:15:58.383114 1581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:15:58.384653 1581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:36269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:38899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:15:58.384899 1581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:15:58.385118 1581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:15:58.389955 1589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:15:58.390084 1586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:15:58.389958 1587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:15:58.390188 1581 server_base.cc:1034] running on GCE node
I20260924 23:15:58.390424 1581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:15:58.390717 1581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:15:58.391870 1581 hybrid_clock.cc:648] HybridClock initialized: now 1790291758391813 us; error 76 us; skew 500 ppm
Sep 24 23:15:58 dist-test-slave-s1hw krb5kdc[1568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291758, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:15:58.394411 1581 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:15:58.394842 1581 webserver.cc:460] Webserver started at http://127.1.4.254:36985/ using document root <none> and password file <none>
I20260924 23:15:58.395033 1581 fs_manager.cc:359] Metadata directory not provided
I20260924 23:15:58.395078 1581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:15:58.395175 1581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:15:58.396231 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "55672a6389954b2686466bb4d6b45f39"
format_stamp: "Formatted at 2026-09-24 23:15:58 on dist-test-slave-s1hw"
server_key: "7b32af992c00e57c5fe1ce2705d3a97f"
server_key_iv: "6092f0d3c49e442597cf3ea5e63eb148"
server_key_version: "encryptionkey@0"
I20260924 23:15:58.396548 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "55672a6389954b2686466bb4d6b45f39"
format_stamp: "Formatted at 2026-09-24 23:15:58 on dist-test-slave-s1hw"
server_key: "7b32af992c00e57c5fe1ce2705d3a97f"
server_key_iv: "6092f0d3c49e442597cf3ea5e63eb148"
server_key_version: "encryptionkey@0"
I20260924 23:15:58.397843 1581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:15:58.398535 1596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:58.398701 1581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:15:58.398785 1581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "55672a6389954b2686466bb4d6b45f39"
format_stamp: "Formatted at 2026-09-24 23:15:58 on dist-test-slave-s1hw"
server_key: "7b32af992c00e57c5fe1ce2705d3a97f"
server_key_iv: "6092f0d3c49e442597cf3ea5e63eb148"
server_key_version: "encryptionkey@0"
I20260924 23:15:58.398846 1581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:15:58.420867 1581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:15:58.423067 1581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:15:58.423205 1581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:15:58.426847 1581 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:38899
I20260924 23:15:58.426947 1648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:38899 every 8 connection(s)
I20260924 23:15:58.427258 1581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:15:58.428188 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:15:58.430514 1649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39: Bootstrap starting.
I20260924 23:15:58.431107 1649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39: Neither blocks nor log segments found. Creating new log.
I20260924 23:15:58.431432 1649 log.cc:824] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39: Log is configured to *not* fsync() on all Append() calls
I20260924 23:15:58.432236 1649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39: No bootstrap required, opened a new log
I20260924 23:15:58.433955 1649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55672a6389954b2686466bb4d6b45f39" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38899 } }
I20260924 23:15:58.434123 1649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:15:58.434152 1649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55672a6389954b2686466bb4d6b45f39, State: Initialized, Role: FOLLOWER
I20260924 23:15:58.434406 1649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55672a6389954b2686466bb4d6b45f39" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38899 } }
I20260924 23:15:58.434508 1649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:15:58.434535 1649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:15:58.434566 1649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:15:58.435185 1649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55672a6389954b2686466bb4d6b45f39" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38899 } }
I20260924 23:15:58.435276 1649 leader_election.cc:302] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55672a6389954b2686466bb4d6b45f39; no voters:
I20260924 23:15:58.435484 1649 leader_election.cc:288] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:15:58.435559 1652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:15:58.435752 1649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:15:58.436038 1652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [term 1 LEADER]: Becoming Leader. State: Replica: 55672a6389954b2686466bb4d6b45f39, State: Running, Role: LEADER
I20260924 23:15:58.436136 1652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55672a6389954b2686466bb4d6b45f39" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38899 } }
I20260924 23:15:58.436524 1653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55672a6389954b2686466bb4d6b45f39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55672a6389954b2686466bb4d6b45f39" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38899 } } }
I20260924 23:15:58.436537 1654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55672a6389954b2686466bb4d6b45f39. Latest consensus state: current_term: 1 leader_uuid: "55672a6389954b2686466bb4d6b45f39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55672a6389954b2686466bb4d6b45f39" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38899 } } }
I20260924 23:15:58.436631 1653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [sys.catalog]: This master's current role is: LEADER
I20260924 23:15:58.436640 1654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39 [sys.catalog]: This master's current role is: LEADER
I20260924 23:15:58.436816 1663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:15:58.437229 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 1581
I20260924 23:15:58.437317 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:15:58.437413 1663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:15:58.437471 1043 external_mini_cluster.cc:1428] Setting key 511885b3062acf5675cbe40d2ff98355
I20260924 23:15:58.438969 1663 catalog_manager.cc:1347] Generated new cluster ID: 5998cf9df5874a6c809aadb11649b4b1
I20260924 23:15:58.439023 1663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:15:58 dist-test-slave-s1hw krb5kdc[1568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:15:58.447264 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:58.437998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.1.4.254:38899)
0924 23:15:58.438143 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:58.438153 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:58.438527 (+ 374us) server_negotiation.cc:407] Connection header received
0924 23:15:58.438668 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:58.438677 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:58.438736 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:58.438948 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:15:58.439712 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.440649 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.441378 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.441817 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.441911 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:58.444171 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:15:58.444175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:15:58.444175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:15:58.444187 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:15:58.445759 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:58.446105 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:58.446109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:58.446109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:58.446205 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:58.446382 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:58.446382 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:58.446383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:58.446652 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:15:58.446805 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:58.446904 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 23:15:58.447056 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260924 23:15:58.452548 1663 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:15:58.453181 1663 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:15:58.462527 1663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55672a6389954b2686466bb4d6b45f39: Generated new TSK 0
I20260924 23:15:58.462721 1663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:15:58.492535 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38899
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:36269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:15:58.572750 1675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:15:58.572970 1675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:15:58.573002 1675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:15:58.574163 1675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:15:58.574208 1675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:15:58.574245 1675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:15:58.575803 1675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:36269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38899
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:15:58.576048 1675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:15:58.576264 1675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:15:58.581248 1681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:15:58.581386 1680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:15:58.581423 1683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:15:58.581275 1675 server_base.cc:1034] running on GCE node
I20260924 23:15:58.581692 1675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:15:58.581929 1675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:15:58.583108 1675 hybrid_clock.cc:648] HybridClock initialized: now 1790291758583092 us; error 34 us; skew 500 ppm
Sep 24 23:15:58 dist-test-slave-s1hw krb5kdc[1568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291758, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:15:58.585775 1675 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:15:58.586273 1675 webserver.cc:460] Webserver started at http://127.1.4.193:46441/ using document root <none> and password file <none>
I20260924 23:15:58.586566 1675 fs_manager.cc:359] Metadata directory not provided
I20260924 23:15:58.586632 1675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:15:58.586767 1675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:15:58.587960 1675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "9493f5cdcd3e42c39d4419603ad65bd8"
format_stamp: "Formatted at 2026-09-24 23:15:58 on dist-test-slave-s1hw"
server_key: "fca92029c701d8866ad184817ed66d85"
server_key_iv: "c1e5d8ef15f2d0a403e061b9aa57c7d4"
server_key_version: "encryptionkey@0"
I20260924 23:15:58.588286 1675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "9493f5cdcd3e42c39d4419603ad65bd8"
format_stamp: "Formatted at 2026-09-24 23:15:58 on dist-test-slave-s1hw"
server_key: "fca92029c701d8866ad184817ed66d85"
server_key_iv: "c1e5d8ef15f2d0a403e061b9aa57c7d4"
server_key_version: "encryptionkey@0"
I20260924 23:15:58.589579 1675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:15:58.590320 1690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:58.590487 1675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:15:58.590576 1675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "9493f5cdcd3e42c39d4419603ad65bd8"
format_stamp: "Formatted at 2026-09-24 23:15:58 on dist-test-slave-s1hw"
server_key: "fca92029c701d8866ad184817ed66d85"
server_key_iv: "c1e5d8ef15f2d0a403e061b9aa57c7d4"
server_key_version: "encryptionkey@0"
I20260924 23:15:58.590651 1675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:15:58.609375 1675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:15:58.611302 1675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:15:58.611479 1675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:15:58.611699 1675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:15:58.611990 1675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:15:58.612022 1675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:58.612051 1675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:15:58.612074 1675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:58.617408 1675 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:41887
I20260924 23:15:58.617460 1803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:41887 every 8 connection(s)
I20260924 23:15:58.617759 1675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:15:58 dist-test-slave-s1hw krb5kdc[1568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291758, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:15:58.626871 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:58.618483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:48157 (local address 127.1.4.254:38899)
0924 23:15:58.618535 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:58.618537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:58.618707 (+ 170us) server_negotiation.cc:407] Connection header received
0924 23:15:58.619737 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:58.619738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:58.619753 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:58.619814 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:15:58.620416 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.620914 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.622021 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.622231 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.622287 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:58.624400 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:15:58.624402 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:15:58.624402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:15:58.624411 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:15:58.625604 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:58.626053 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:58.626054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:58.626055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:58.626109 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:58.626354 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:58.626354 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:58.626354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:58.626473 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:15:58.626522 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:58.626737 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 23:15:58.626804 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 23:15:58.627084 1806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:58.618416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38899 (local address 127.1.4.193:48157)
0924 23:15:58.618575 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 23:15:58.618617 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:15:58.619621 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:15:58.619876 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:15:58.619879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:15:58.620134 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:15:58.620328 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:58.620333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.621045 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.621046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:58.621879 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:58.621881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.622295 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.622296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:58.622401 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:58.622867 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:15:58.622875 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:15:58.624307 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:15:58.625679 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:15:58.625684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:15:58.625686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:15:58.625971 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:15:58.626189 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:15:58.626190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:15:58.626190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:15:58.626301 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:15:58.626521 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:15:58.626524 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:15:58.626640 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 23:15:58.626717 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 23:15:58.626845 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260924 23:15:58.627615 1804 heartbeater.cc:347] Connected to a master server at 127.1.4.254:38899
I20260924 23:15:58.627717 1804 heartbeater.cc:464] Registering TS with master...
I20260924 23:15:58.627763 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 1675
I20260924 23:15:58.627853 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:15:58.627902 1804 heartbeater.cc:511] Master 127.1.4.254:38899 requested a full tablet report, sending...
I20260924 23:15:58.628108 1043 external_mini_cluster.cc:1428] Setting key d6830a03ed2bf2ac40fbaeab54fc47af
I20260924 23:15:58.628440 1613 ts_manager.cc:194] Registered new tserver with Master: 9493f5cdcd3e42c39d4419603ad65bd8 (127.1.4.193:41887)
I20260924 23:15:58.629206 1613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:48157
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:15:58.652179 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38899
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:36269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:15:58.736564 1809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:15:58.736784 1809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:15:58.736819 1809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:15:58.738026 1809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:15:58.738071 1809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:15:58.738133 1809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:15:58.739693 1809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:36269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38899
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:15:58.739935 1809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:15:58.740137 1809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:15:58.745319 1814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:15:58.745357 1817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:15:58.745412 1809 server_base.cc:1034] running on GCE node
W20260924 23:15:58.745357 1815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:15:58.745731 1809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:15:58.745952 1809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:15:58.747114 1809 hybrid_clock.cc:648] HybridClock initialized: now 1790291758747099 us; error 30 us; skew 500 ppm
Sep 24 23:15:58 dist-test-slave-s1hw krb5kdc[1568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291758, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:15:58.749776 1809 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:15:58.750226 1809 webserver.cc:460] Webserver started at http://127.1.4.194:41167/ using document root <none> and password file <none>
I20260924 23:15:58.750437 1809 fs_manager.cc:359] Metadata directory not provided
I20260924 23:15:58.750480 1809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:15:58.750597 1809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:15:58.751667 1809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "3b737b1b6f6d4dd281a66de02ef4c1a8"
format_stamp: "Formatted at 2026-09-24 23:15:58 on dist-test-slave-s1hw"
server_key: "1eaa1fb272e07b3a4e6cb4a6eaf69748"
server_key_iv: "b2166fbaf8a50eee2eab27b893f5c7bf"
server_key_version: "encryptionkey@0"
I20260924 23:15:58.751989 1809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "3b737b1b6f6d4dd281a66de02ef4c1a8"
format_stamp: "Formatted at 2026-09-24 23:15:58 on dist-test-slave-s1hw"
server_key: "1eaa1fb272e07b3a4e6cb4a6eaf69748"
server_key_iv: "b2166fbaf8a50eee2eab27b893f5c7bf"
server_key_version: "encryptionkey@0"
I20260924 23:15:58.753229 1809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:15:58.753921 1824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:58.754112 1809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:15:58.754194 1809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "3b737b1b6f6d4dd281a66de02ef4c1a8"
format_stamp: "Formatted at 2026-09-24 23:15:58 on dist-test-slave-s1hw"
server_key: "1eaa1fb272e07b3a4e6cb4a6eaf69748"
server_key_iv: "b2166fbaf8a50eee2eab27b893f5c7bf"
server_key_version: "encryptionkey@0"
I20260924 23:15:58.754247 1809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:15:58.765750 1809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:15:58.768366 1809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:15:58.768529 1809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:15:58.768751 1809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:15:58.769115 1809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:15:58.769171 1809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:58.769212 1809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:15:58.769238 1809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:58.775792 1809 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:38439
I20260924 23:15:58.775856 1937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:38439 every 8 connection(s)
I20260924 23:15:58.776255 1809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:15:58.776613 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 1809
I20260924 23:15:58.776705 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:15:58.776880 1043 external_mini_cluster.cc:1428] Setting key 3480359858ca511064469e8cc0dcbd62
Sep 24 23:15:58 dist-test-slave-s1hw krb5kdc[1568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291758, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
I20260924 23:15:58.788316 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:58.777489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:52915 (local address 127.1.4.254:38899)
0924 23:15:58.777549 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:58.777550 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:58.777567 (+ 17us) server_negotiation.cc:407] Connection header received
0924 23:15:58.778731 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:58.778732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:58.778746 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:58.778798 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:15:58.779655 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.780377 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.781860 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.782174 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.782253 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:58.785045 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:15:58.785047 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:15:58.785048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:15:58.785057 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:15:58.786713 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:58.787225 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:58.787227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:58.787227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:58.787286 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:58.787601 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:58.787602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:58.787602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:58.787748 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:15:58.787810 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:58.788139 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 23:15:58.788222 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
I20260924 23:15:58.788528 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:58.777188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38899 (local address 127.1.4.194:52915)
0924 23:15:58.777373 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 23:15:58.777421 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 23:15:58.778626 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:15:58.778878 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:15:58.778883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:15:58.779254 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:15:58.779560 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:58.779566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.780486 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.780487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:58.781778 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:58.781781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.782286 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.782287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:58.782445 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:58.783105 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:15:58.783115 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:15:58.784904 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:15:58.786808 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:15:58.786817 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:15:58.786820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:15:58.787134 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:15:58.787402 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:15:58.787403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:15:58.787403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:15:58.787534 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:15:58.787814 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:15:58.787851 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:15:58.788002 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 23:15:58.788088 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 23:15:58.788286 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260924 23:15:58.789029 1938 heartbeater.cc:347] Connected to a master server at 127.1.4.254:38899
I20260924 23:15:58.789144 1938 heartbeater.cc:464] Registering TS with master...
I20260924 23:15:58.789345 1938 heartbeater.cc:511] Master 127.1.4.254:38899 requested a full tablet report, sending...
I20260924 23:15:58.789760 1613 ts_manager.cc:194] Registered new tserver with Master: 3b737b1b6f6d4dd281a66de02ef4c1a8 (127.1.4.194:38439)
I20260924 23:15:58.790258 1613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:52915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:15:58.806547 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38899
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:36269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:15:58.887475 1943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:15:58.887681 1943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:15:58.887705 1943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:15:58.888885 1943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:15:58.888924 1943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:15:58.888957 1943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:15:58.890410 1943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:36269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38899
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:15:58.890645 1943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:15:58.890836 1943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:15:58.896095 1949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:15:58.896152 1951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:15:58.896118 1948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:15:58.896173 1943 server_base.cc:1034] running on GCE node
I20260924 23:15:58.896571 1943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:15:58.896776 1943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:15:58.897927 1943 hybrid_clock.cc:648] HybridClock initialized: now 1790291758897909 us; error 31 us; skew 500 ppm
Sep 24 23:15:58 dist-test-slave-s1hw krb5kdc[1568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291758, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:15:58.900730 1943 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:15:58.901242 1943 webserver.cc:460] Webserver started at http://127.1.4.195:35691/ using document root <none> and password file <none>
I20260924 23:15:58.901464 1943 fs_manager.cc:359] Metadata directory not provided
I20260924 23:15:58.901525 1943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:15:58.901655 1943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:15:58.902818 1943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "8341e8274bdb4b75b1ace4b7b505318c"
format_stamp: "Formatted at 2026-09-24 23:15:58 on dist-test-slave-s1hw"
server_key: "d6897192e455a35b87f764cc197f2208"
server_key_iv: "be6be77ea7914f28b1b095c73a01e2f4"
server_key_version: "encryptionkey@0"
I20260924 23:15:58.903173 1943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "8341e8274bdb4b75b1ace4b7b505318c"
format_stamp: "Formatted at 2026-09-24 23:15:58 on dist-test-slave-s1hw"
server_key: "d6897192e455a35b87f764cc197f2208"
server_key_iv: "be6be77ea7914f28b1b095c73a01e2f4"
server_key_version: "encryptionkey@0"
I20260924 23:15:58.904527 1943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:15:58.905303 1958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:58.905468 1943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:15:58.905544 1943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "8341e8274bdb4b75b1ace4b7b505318c"
format_stamp: "Formatted at 2026-09-24 23:15:58 on dist-test-slave-s1hw"
server_key: "d6897192e455a35b87f764cc197f2208"
server_key_iv: "be6be77ea7914f28b1b095c73a01e2f4"
server_key_version: "encryptionkey@0"
I20260924 23:15:58.905612 1943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:15:58.917344 1943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:15:58.919230 1943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:15:58.919350 1943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:15:58.919593 1943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:15:58.919868 1943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:15:58.919899 1943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:58.919921 1943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:15:58.919945 1943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:15:58.925541 1943 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:46047
I20260924 23:15:58.925596 2071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:46047 every 8 connection(s)
I20260924 23:15:58.925889 1943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:15:58.931139 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 1943
I20260924 23:15:58.931262 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:15:58.931514 1043 external_mini_cluster.cc:1428] Setting key fca35bb8ce7f8971addd4ee633550822
Sep 24 23:15:58 dist-test-slave-s1hw krb5kdc[1568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291758, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:15:58.936362 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:58.926613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:42799 (local address 127.1.4.254:38899)
0924 23:15:58.926680 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:58.926682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:58.926836 (+ 154us) server_negotiation.cc:407] Connection header received
0924 23:15:58.927840 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:58.927841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:58.927852 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:58.927960 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:15:58.928596 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.929109 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.930263 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.930457 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.930522 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:58.933182 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:15:58.933183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:15:58.933183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:15:58.933190 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:15:58.934736 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:58.935219 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:58.935220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:58.935221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:58.935277 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:58.935608 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:58.935609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:58.935609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:58.935729 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:15:58.935786 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:58.936221 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 23:15:58.936294 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 23:15:58.936586 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:58.926570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38899 (local address 127.1.4.195:42799)
0924 23:15:58.926711 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 23:15:58.926751 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:15:58.927719 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:15:58.928027 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:15:58.928030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:15:58.928296 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:15:58.928490 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:58.928495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.929216 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.929217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:58.930136 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:58.930138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.930541 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.930542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:58.930670 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:58.931247 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:15:58.931259 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:15:58.933079 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:15:58.934828 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:15:58.934835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:15:58.934838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:15:58.935141 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:15:58.935395 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:15:58.935397 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:15:58.935397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:15:58.935526 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:15:58.935818 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:15:58.935826 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:15:58.936009 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 23:15:58.936110 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 23:15:58.936269 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260924 23:15:58.937120 2072 heartbeater.cc:347] Connected to a master server at 127.1.4.254:38899
I20260924 23:15:58.937235 2072 heartbeater.cc:464] Registering TS with master...
I20260924 23:15:58.937417 2072 heartbeater.cc:511] Master 127.1.4.254:38899 requested a full tablet report, sending...
I20260924 23:15:58.937774 1613 ts_manager.cc:194] Registered new tserver with Master: 8341e8274bdb4b75b1ace4b7b505318c (127.1.4.195:46047)
I20260924 23:15:58.938215 1613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:42799
I20260924 23:15:58.943315 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:15:58.950723 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:58.945001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42816 (local address 127.1.4.254:38899)
0924 23:15:58.945052 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:58.945054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:58.945082 (+ 28us) server_negotiation.cc:407] Connection header received
0924 23:15:58.945140 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:58.945140 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:58.945149 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:58.945203 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:15:58.945728 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.946253 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.947113 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.947317 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.947424 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:58.948550 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:15:58.948551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:15:58.948551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:15:58.948558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:15:58.949788 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:58.950053 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:58.950054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:58.950054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:58.950103 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:58.950222 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:58.950223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:58.950223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:58.950353 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:15:58.950423 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:58.950583 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 23:15:58.950655 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 23:15:58.952984 1613 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:15:58.954567 1613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:15:58.961359 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:58.956816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:41887 (local address 127.0.0.1:47704)
0924 23:15:58.957210 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0924 23:15:58.957218 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:15:58.957251 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:15:58.957494 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:15:58.957494 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:15:58.957496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:15:58.957527 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:58.957528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.958710 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.958711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:58.960120 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:58.960122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.961056 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.961057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:58.961121 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:58.961133 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 23:15:58.961215 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 23:15:58.961273 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":108,"spinlock_wait_cycles":768,"thread_start_us":63,"threads_started":1}
I20260924 23:15:58.961681 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:58.957182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47704 (local address 127.1.4.193:41887)
0924 23:15:58.957352 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:58.957357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:58.957365 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:15:58.957398 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:58.957400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:58.957414 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:58.957486 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:15:58.957567 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.958636 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.960196 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.960933 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.961007 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:58.961061 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:58.961551 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 23:15:58.961596 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
I20260924 23:15:58.962114 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:58.956816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:46047 (local address 127.0.0.1:52450)
0924 23:15:58.956995 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 23:15:58.957037 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:15:58.957108 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:15:58.957342 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:15:58.957344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:15:58.957346 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:15:58.957410 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:58.957411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.958703 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.958704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:58.960119 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:58.960122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.961883 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.961884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:58.961957 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:58.961962 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:15:58.962006 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:15:58.962045 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":5,"thread_start_us":67,"threads_started":1}
I20260924 23:15:58.962280 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:58.957016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:38439 (local address 127.0.0.1:33238)
0924 23:15:58.958750 (+ 1734us) negotiation.cc:107] Waiting for socket to connect
0924 23:15:58.958759 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:15:58.958809 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:15:58.959014 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:15:58.959014 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:15:58.959016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:15:58.959055 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:58.959056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.960050 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.960051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:58.961117 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:58.961119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.962064 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.962065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:58.962124 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:58.962128 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:15:58.962180 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 23:15:58.962218 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1721,"thread_start_us":56,"threads_started":1}
I20260924 23:15:58.962328 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:58.956985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.1.4.195:46047)
0924 23:15:58.957184 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:58.957189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:58.957201 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:15:58.957241 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:58.957243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:58.957259 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:58.957341 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:15:58.957460 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.958610 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.961099 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.961726 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.961794 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:58.961851 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:58.962214 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 23:15:58.962258 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":90,"threads_started":1}
I20260924 23:15:58.962414 1738 tablet_service.cc:1467] Processing CreateTablet for tablet cac131caecbf458994b4f061b12034d4 (DEFAULT_TABLE table=test-table [id=df1e67e8fe674982b4c202064d91f93c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:15:58.962730 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:58.957141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33238 (local address 127.1.4.194:38439)
0924 23:15:58.957316 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:58.957320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:58.958825 (+ 1505us) server_negotiation.cc:407] Connection header received
0924 23:15:58.958869 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:58.958871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:58.958886 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:58.958948 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:15:58.959119 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.959971 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.961201 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:58.961906 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:58.961975 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:58.962031 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:58.962632 (+ 601us) server_negotiation.cc:299] Negotiation successful
0924 23:15:58.962674 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260924 23:15:58.962749 1738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cac131caecbf458994b4f061b12034d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:15:58.962922 2006 tablet_service.cc:1467] Processing CreateTablet for tablet cac131caecbf458994b4f061b12034d4 (DEFAULT_TABLE table=test-table [id=df1e67e8fe674982b4c202064d91f93c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:15:58.963218 2006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cac131caecbf458994b4f061b12034d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:15:58.963256 1872 tablet_service.cc:1467] Processing CreateTablet for tablet cac131caecbf458994b4f061b12034d4 (DEFAULT_TABLE table=test-table [id=df1e67e8fe674982b4c202064d91f93c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:15:58.963526 1872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cac131caecbf458994b4f061b12034d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:15:58.966085 2086 tablet_bootstrap.cc:492] T cac131caecbf458994b4f061b12034d4 P 3b737b1b6f6d4dd281a66de02ef4c1a8: Bootstrap starting.
I20260924 23:15:58.966269 2087 tablet_bootstrap.cc:492] T cac131caecbf458994b4f061b12034d4 P 9493f5cdcd3e42c39d4419603ad65bd8: Bootstrap starting.
I20260924 23:15:58.966295 2088 tablet_bootstrap.cc:492] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c: Bootstrap starting.
I20260924 23:15:58.966787 2087 tablet_bootstrap.cc:654] T cac131caecbf458994b4f061b12034d4 P 9493f5cdcd3e42c39d4419603ad65bd8: Neither blocks nor log segments found. Creating new log.
I20260924 23:15:58.966785 2086 tablet_bootstrap.cc:654] T cac131caecbf458994b4f061b12034d4 P 3b737b1b6f6d4dd281a66de02ef4c1a8: Neither blocks nor log segments found. Creating new log.
I20260924 23:15:58.966835 2088 tablet_bootstrap.cc:654] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c: Neither blocks nor log segments found. Creating new log.
I20260924 23:15:58.967234 2087 log.cc:824] T cac131caecbf458994b4f061b12034d4 P 9493f5cdcd3e42c39d4419603ad65bd8: Log is configured to *not* fsync() on all Append() calls
I20260924 23:15:58.967331 2086 log.cc:824] T cac131caecbf458994b4f061b12034d4 P 3b737b1b6f6d4dd281a66de02ef4c1a8: Log is configured to *not* fsync() on all Append() calls
I20260924 23:15:58.967401 2088 log.cc:824] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c: Log is configured to *not* fsync() on all Append() calls
I20260924 23:15:58.968000 2086 tablet_bootstrap.cc:492] T cac131caecbf458994b4f061b12034d4 P 3b737b1b6f6d4dd281a66de02ef4c1a8: No bootstrap required, opened a new log
I20260924 23:15:58.968005 2087 tablet_bootstrap.cc:492] T cac131caecbf458994b4f061b12034d4 P 9493f5cdcd3e42c39d4419603ad65bd8: No bootstrap required, opened a new log
I20260924 23:15:58.968014 2088 tablet_bootstrap.cc:492] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c: No bootstrap required, opened a new log
I20260924 23:15:58.968060 2086 ts_tablet_manager.cc:1397] T cac131caecbf458994b4f061b12034d4 P 3b737b1b6f6d4dd281a66de02ef4c1a8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:15:58.968062 2087 ts_tablet_manager.cc:1397] T cac131caecbf458994b4f061b12034d4 P 9493f5cdcd3e42c39d4419603ad65bd8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 23:15:58.968214 2088 ts_tablet_manager.cc:1397] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:15:58.969887 2086 raft_consensus.cc:348] T cac131caecbf458994b4f061b12034d4 P 3b737b1b6f6d4dd281a66de02ef4c1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8341e8274bdb4b75b1ace4b7b505318c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46047 } } peers { permanent_uuid: "9493f5cdcd3e42c39d4419603ad65bd8" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 41887 } } peers { permanent_uuid: "3b737b1b6f6d4dd281a66de02ef4c1a8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38439 } }
I20260924 23:15:58.970043 2086 raft_consensus.cc:374] T cac131caecbf458994b4f061b12034d4 P 3b737b1b6f6d4dd281a66de02ef4c1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:15:58.970077 2086 raft_consensus.cc:729] T cac131caecbf458994b4f061b12034d4 P 3b737b1b6f6d4dd281a66de02ef4c1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b737b1b6f6d4dd281a66de02ef4c1a8, State: Initialized, Role: FOLLOWER
I20260924 23:15:58.970191 2086 consensus_queue.cc:260] T cac131caecbf458994b4f061b12034d4 P 3b737b1b6f6d4dd281a66de02ef4c1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8341e8274bdb4b75b1ace4b7b505318c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46047 } } peers { permanent_uuid: "9493f5cdcd3e42c39d4419603ad65bd8" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 41887 } } peers { permanent_uuid: "3b737b1b6f6d4dd281a66de02ef4c1a8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38439 } }
I20260924 23:15:58.970435 2086 ts_tablet_manager.cc:1428] T cac131caecbf458994b4f061b12034d4 P 3b737b1b6f6d4dd281a66de02ef4c1a8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:15:58.970374 2087 raft_consensus.cc:348] T cac131caecbf458994b4f061b12034d4 P 9493f5cdcd3e42c39d4419603ad65bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8341e8274bdb4b75b1ace4b7b505318c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46047 } } peers { permanent_uuid: "9493f5cdcd3e42c39d4419603ad65bd8" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 41887 } } peers { permanent_uuid: "3b737b1b6f6d4dd281a66de02ef4c1a8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38439 } }
I20260924 23:15:58.970374 2088 raft_consensus.cc:348] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8341e8274bdb4b75b1ace4b7b505318c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46047 } } peers { permanent_uuid: "9493f5cdcd3e42c39d4419603ad65bd8" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 41887 } } peers { permanent_uuid: "3b737b1b6f6d4dd281a66de02ef4c1a8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38439 } }
I20260924 23:15:58.970534 2088 raft_consensus.cc:374] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:15:58.970534 2087 raft_consensus.cc:374] T cac131caecbf458994b4f061b12034d4 P 9493f5cdcd3e42c39d4419603ad65bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:15:58.970570 1938 heartbeater.cc:503] Master 127.1.4.254:38899 was elected leader, sending a full tablet report...
I20260924 23:15:58.970574 2088 raft_consensus.cc:729] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8341e8274bdb4b75b1ace4b7b505318c, State: Initialized, Role: FOLLOWER
I20260924 23:15:58.970574 2087 raft_consensus.cc:729] T cac131caecbf458994b4f061b12034d4 P 9493f5cdcd3e42c39d4419603ad65bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9493f5cdcd3e42c39d4419603ad65bd8, State: Initialized, Role: FOLLOWER
I20260924 23:15:58.970693 2088 consensus_queue.cc:260] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8341e8274bdb4b75b1ace4b7b505318c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46047 } } peers { permanent_uuid: "9493f5cdcd3e42c39d4419603ad65bd8" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 41887 } } peers { permanent_uuid: "3b737b1b6f6d4dd281a66de02ef4c1a8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38439 } }
I20260924 23:15:58.970697 2087 consensus_queue.cc:260] T cac131caecbf458994b4f061b12034d4 P 9493f5cdcd3e42c39d4419603ad65bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8341e8274bdb4b75b1ace4b7b505318c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46047 } } peers { permanent_uuid: "9493f5cdcd3e42c39d4419603ad65bd8" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 41887 } } peers { permanent_uuid: "3b737b1b6f6d4dd281a66de02ef4c1a8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38439 } }
I20260924 23:15:58.970947 2088 ts_tablet_manager.cc:1428] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 23:15:58.970947 2087 ts_tablet_manager.cc:1428] T cac131caecbf458994b4f061b12034d4 P 9493f5cdcd3e42c39d4419603ad65bd8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 23:15:58.971064 2072 heartbeater.cc:503] Master 127.1.4.254:38899 was elected leader, sending a full tablet report...
I20260924 23:15:58.971098 1804 heartbeater.cc:503] Master 127.1.4.254:38899 was elected leader, sending a full tablet report...
W20260924 23:15:59.026856 1939 tablet.cc:2369] T cac131caecbf458994b4f061b12034d4 P 3b737b1b6f6d4dd281a66de02ef4c1a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:15:59.075599 2093 raft_consensus.cc:482] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:15:59.075735 2093 raft_consensus.cc:504] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8341e8274bdb4b75b1ace4b7b505318c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46047 } } peers { permanent_uuid: "9493f5cdcd3e42c39d4419603ad65bd8" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 41887 } } peers { permanent_uuid: "3b737b1b6f6d4dd281a66de02ef4c1a8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38439 } }
I20260924 23:15:59.076074 2093 leader_election.cc:288] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9493f5cdcd3e42c39d4419603ad65bd8 (127.1.4.193:41887), 3b737b1b6f6d4dd281a66de02ef4c1a8 (127.1.4.194:38439)
I20260924 23:15:59.078706 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:59.076270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:41887 (local address 127.1.4.195:45371)
0924 23:15:59.076324 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 23:15:59.076337 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:15:59.076398 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:15:59.076529 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:15:59.076530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:15:59.076531 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:15:59.076608 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:59.076609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:59.077213 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:59.077214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:59.077955 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:59.077957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:59.078475 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:59.078475 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:59.078543 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:59.078546 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:15:59.078591 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 23:15:59.078628 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 23:15:59.078706 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:59.076384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:45371 (local address 127.1.4.193:41887)
0924 23:15:59.076423 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:59.076425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:59.076434 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:15:59.076462 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:59.076463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:59.076474 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:59.076525 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:15:59.076671 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:59.077139 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:59.078042 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:59.078400 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:59.078457 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:59.078469 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:59.078607 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 23:15:59.078645 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:15:59.078852 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:59.076389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:58751 (local address 127.1.4.194:38439)
0924 23:15:59.076441 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:59.076444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:59.076552 (+ 108us) server_negotiation.cc:407] Connection header received
0924 23:15:59.076605 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:59.076606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:59.076618 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:59.076659 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:15:59.076796 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:59.077292 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:59.078207 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:59.078559 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:59.078617 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:59.078626 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:59.078758 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 23:15:59.078790 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 23:15:59.078871 2095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:59.076270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:38439 (local address 127.1.4.195:58751)
0924 23:15:59.076479 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 23:15:59.076490 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:15:59.076546 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:15:59.076674 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:15:59.076674 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:15:59.076676 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:15:59.076738 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:59.076739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:59.077375 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:59.077375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:59.078124 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:15:59.078126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:15:59.078634 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:15:59.078635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:15:59.078697 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:59.078700 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:15:59.078743 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 23:15:59.078780 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":16,"thread_start_us":105,"threads_started":1}
I20260924 23:15:59.079041 1758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cac131caecbf458994b4f061b12034d4" candidate_uuid: "8341e8274bdb4b75b1ace4b7b505318c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9493f5cdcd3e42c39d4419603ad65bd8" is_pre_election: true
I20260924 23:15:59.079167 1892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cac131caecbf458994b4f061b12034d4" candidate_uuid: "8341e8274bdb4b75b1ace4b7b505318c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b737b1b6f6d4dd281a66de02ef4c1a8" is_pre_election: true
I20260924 23:15:59.079205 1758 raft_consensus.cc:2454] T cac131caecbf458994b4f061b12034d4 P 9493f5cdcd3e42c39d4419603ad65bd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8341e8274bdb4b75b1ace4b7b505318c in term 0.
I20260924 23:15:59.079289 1892 raft_consensus.cc:2454] T cac131caecbf458994b4f061b12034d4 P 3b737b1b6f6d4dd281a66de02ef4c1a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8341e8274bdb4b75b1ace4b7b505318c in term 0.
I20260924 23:15:59.079452 1962 leader_election.cc:302] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8341e8274bdb4b75b1ace4b7b505318c, 9493f5cdcd3e42c39d4419603ad65bd8; no voters:
I20260924 23:15:59.079599 2093 raft_consensus.cc:2781] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:15:59.079668 2093 raft_consensus.cc:482] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:15:59.079684 2093 raft_consensus.cc:3037] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:15:59.080369 2093 raft_consensus.cc:504] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8341e8274bdb4b75b1ace4b7b505318c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46047 } } peers { permanent_uuid: "9493f5cdcd3e42c39d4419603ad65bd8" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 41887 } } peers { permanent_uuid: "3b737b1b6f6d4dd281a66de02ef4c1a8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38439 } }
I20260924 23:15:59.080564 2093 leader_election.cc:288] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [CANDIDATE]: Term 1 election: Requested vote from peers 9493f5cdcd3e42c39d4419603ad65bd8 (127.1.4.193:41887), 3b737b1b6f6d4dd281a66de02ef4c1a8 (127.1.4.194:38439)
I20260924 23:15:59.080665 1892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cac131caecbf458994b4f061b12034d4" candidate_uuid: "8341e8274bdb4b75b1ace4b7b505318c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b737b1b6f6d4dd281a66de02ef4c1a8"
I20260924 23:15:59.080729 1758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cac131caecbf458994b4f061b12034d4" candidate_uuid: "8341e8274bdb4b75b1ace4b7b505318c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9493f5cdcd3e42c39d4419603ad65bd8"
I20260924 23:15:59.080760 1892 raft_consensus.cc:3037] T cac131caecbf458994b4f061b12034d4 P 3b737b1b6f6d4dd281a66de02ef4c1a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:15:59.080801 1758 raft_consensus.cc:3037] T cac131caecbf458994b4f061b12034d4 P 9493f5cdcd3e42c39d4419603ad65bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:15:59.081440 1758 raft_consensus.cc:2454] T cac131caecbf458994b4f061b12034d4 P 9493f5cdcd3e42c39d4419603ad65bd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8341e8274bdb4b75b1ace4b7b505318c in term 1.
I20260924 23:15:59.081513 1892 raft_consensus.cc:2454] T cac131caecbf458994b4f061b12034d4 P 3b737b1b6f6d4dd281a66de02ef4c1a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8341e8274bdb4b75b1ace4b7b505318c in term 1.
I20260924 23:15:59.081715 1961 leader_election.cc:302] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b737b1b6f6d4dd281a66de02ef4c1a8, 8341e8274bdb4b75b1ace4b7b505318c; no voters:
I20260924 23:15:59.081813 2093 raft_consensus.cc:2781] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:15:59.081995 2093 raft_consensus.cc:686] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [term 1 LEADER]: Becoming Leader. State: Replica: 8341e8274bdb4b75b1ace4b7b505318c, State: Running, Role: LEADER
I20260924 23:15:59.082114 2093 consensus_queue.cc:237] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8341e8274bdb4b75b1ace4b7b505318c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46047 } } peers { permanent_uuid: "9493f5cdcd3e42c39d4419603ad65bd8" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 41887 } } peers { permanent_uuid: "3b737b1b6f6d4dd281a66de02ef4c1a8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38439 } }
I20260924 23:15:59.083026 1612 catalog_manager.cc:5495] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8341e8274bdb4b75b1ace4b7b505318c (127.1.4.195). New cstate: current_term: 1 leader_uuid: "8341e8274bdb4b75b1ace4b7b505318c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8341e8274bdb4b75b1ace4b7b505318c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9493f5cdcd3e42c39d4419603ad65bd8" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 41887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b737b1b6f6d4dd281a66de02ef4c1a8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38439 } health_report { overall_health: UNKNOWN } } }
Sep 24 23:15:59 dist-test-slave-s1hw krb5kdc[1568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.193@KRBTEST.COM
I20260924 23:15:59.110152 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:59.103032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.1.4.193:41887)
0924 23:15:59.103078 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:59.103081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:59.103180 (+ 99us) server_negotiation.cc:407] Connection header received
0924 23:15:59.103245 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:59.103246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:59.103257 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:59.103302 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:15:59.103871 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:59.104344 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:59.105045 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:59.105233 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:59.105289 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:59.107606 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:15:59.107617 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:15:59.107617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:15:59.107625 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:15:59.109153 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:59.109552 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:59.109555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:59.109555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:59.109609 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:59.109744 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:59.109744 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:59.109745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:59.109871 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:15:59.109986 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:59.110018 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 23:15:59.110079 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 23:15:59.118512 1805 tablet.cc:2369] T cac131caecbf458994b4f061b12034d4 P 9493f5cdcd3e42c39d4419603ad65bd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 23:15:59 dist-test-slave-s1hw krb5kdc[1568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291759, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 23:15:59 dist-test-slave-s1hw krb5kdc[1568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291759, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.4.193@KRBTEST.COM
I20260924 23:15:59.154008 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:59.146002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47714 (local address 127.1.4.193:41887)
0924 23:15:59.146064 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:59.146066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:59.146243 (+ 177us) server_negotiation.cc:407] Connection header received
0924 23:15:59.146279 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:59.146280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:59.146291 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:59.146351 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:15:59.147086 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:59.147636 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:59.148688 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:59.148880 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:59.148942 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:59.151634 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:15:59.151636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:15:59.151636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:15:59.151646 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:15:59.152932 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:59.153267 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:59.153268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:59.153269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:59.153317 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:59.153485 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:59.153486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:59.153487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:59.153583 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:15:59.153666 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:59.153838 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 23:15:59.153913 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 23:15:59 dist-test-slave-s1hw krb5kdc[1568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260924 23:15:59.176465 2073 tablet.cc:2369] T cac131caecbf458994b4f061b12034d4 P 8341e8274bdb4b75b1ace4b7b505318c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 23:15:59 dist-test-slave-s1hw krb5kdc[1568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.193@KRBTEST.COM
I20260924 23:15:59.215240 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:15:59.207633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47724 (local address 127.1.4.193:41887)
0924 23:15:59.207694 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 23:15:59.207696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:15:59.207790 (+ 94us) server_negotiation.cc:407] Connection header received
0924 23:15:59.207856 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:15:59.207857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:15:59.207866 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:15:59.207994 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:15:59.208692 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:59.209230 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:59.210281 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:15:59.210485 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:15:59.210546 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:15:59.212907 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:15:59.212908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:15:59.212908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:15:59.212915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:15:59.214219 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:59.214548 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:59.214549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:59.214549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:59.214596 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:15:59.214737 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:15:59.214737 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:15:59.214738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:15:59.214822 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:15:59.214884 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 23:15:59.215107 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 23:15:59.215169 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 23:15:59.215706 1718 tablet_service.cc:2908] Rejecting scan request for tablet cac131caecbf458994b4f061b12034d4: Uninitialized: clean time has not yet been initialized
I20260924 23:15:59.216646 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 1675
I20260924 23:15:59.220818 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 1809
I20260924 23:15:59.225224 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 1943
I20260924 23:15:59.228973 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 1581
2026-09-24T23:15:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2001 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:15:59 dist-test-slave-s1hw krb5kdc[2110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:15:59 dist-test-slave-s1hw krb5kdc[2110](info): set up 2 sockets
Sep 24 23:15:59 dist-test-slave-s1hw krb5kdc[2110](info): commencing operation
krb5kdc: starting...
W20260924 23:16:00.250849 1043 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:16:00 dist-test-slave-s1hw krb5kdc[2110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:16:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:16:00.368521 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:42509
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:42509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:00.450927 2123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:00.451179 2123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:00.451205 2123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:00.452554 2123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:00.452615 2123 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 23:16:00.452627 2123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:00.452641 2123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:00.452651 2123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:00.452663 2123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:00.454391 2123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:42509
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:42509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:00.454676 2123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:00.454917 2123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:00.459853 2128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:00.459990 2129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:00.459875 2131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:00.460501 2123 server_base.cc:1034] running on GCE node
I20260924 23:16:00.460816 2123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:00.461145 2123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:00.462332 2123 hybrid_clock.cc:648] HybridClock initialized: now 1790291760462315 us; error 31 us; skew 500 ppm
Sep 24 23:16:00 dist-test-slave-s1hw krb5kdc[2110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291760, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:00.465309 2123 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:16:00.465821 2123 webserver.cc:460] Webserver started at http://127.1.4.254:44237/ using document root <none> and password file <none>
I20260924 23:16:00.466041 2123 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:00.466092 2123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:00.466238 2123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:00.467404 2123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "b5d00a6e848c43db884e9b89271b47a0"
format_stamp: "Formatted at 2026-09-24 23:16:00 on dist-test-slave-s1hw"
server_key: "e9eb3329022355b0e6a6fa7c8ad9963b"
server_key_iv: "a78eb8e99d36878b3da7fb3caa574bac"
server_key_version: "encryptionkey@0"
I20260924 23:16:00.467767 2123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "b5d00a6e848c43db884e9b89271b47a0"
format_stamp: "Formatted at 2026-09-24 23:16:00 on dist-test-slave-s1hw"
server_key: "e9eb3329022355b0e6a6fa7c8ad9963b"
server_key_iv: "a78eb8e99d36878b3da7fb3caa574bac"
server_key_version: "encryptionkey@0"
I20260924 23:16:00.469128 2123 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:00.469986 2138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:00.470198 2123 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:00.470299 2123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "b5d00a6e848c43db884e9b89271b47a0"
format_stamp: "Formatted at 2026-09-24 23:16:00 on dist-test-slave-s1hw"
server_key: "e9eb3329022355b0e6a6fa7c8ad9963b"
server_key_iv: "a78eb8e99d36878b3da7fb3caa574bac"
server_key_version: "encryptionkey@0"
I20260924 23:16:00.470371 2123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:00.493683 2123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:00.495983 2123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:00.496133 2123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:00.500919 2123 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:42509
I20260924 23:16:00.501000 2200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:42509 every 8 connection(s)
I20260924 23:16:00.501318 2123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:00.501993 2201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:00.503700 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 2123
I20260924 23:16:00.503777 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:00.503923 1043 external_mini_cluster.cc:1428] Setting key c3c1190328097f9acc8cd056a0f3bc11
I20260924 23:16:00.504626 2201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0: Bootstrap starting.
I20260924 23:16:00.505319 2201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:00.505653 2201 log.cc:824] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:00.506500 2201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0: No bootstrap required, opened a new log
I20260924 23:16:00.509107 2201 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d00a6e848c43db884e9b89271b47a0" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42509 } }
I20260924 23:16:00.509317 2201 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:00.509369 2201 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d00a6e848c43db884e9b89271b47a0, State: Initialized, Role: FOLLOWER
I20260924 23:16:00.509531 2201 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d00a6e848c43db884e9b89271b47a0" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42509 } }
I20260924 23:16:00.509624 2201 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:00.509668 2201 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:00.509722 2201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:00.510397 2201 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d00a6e848c43db884e9b89271b47a0" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42509 } }
I20260924 23:16:00.510496 2201 leader_election.cc:302] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5d00a6e848c43db884e9b89271b47a0; no voters:
I20260924 23:16:00.510673 2201 leader_election.cc:288] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:00.510722 2206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:00.510886 2206 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [term 1 LEADER]: Becoming Leader. State: Replica: b5d00a6e848c43db884e9b89271b47a0, State: Running, Role: LEADER
I20260924 23:16:00.510936 2201 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:00.510979 2206 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d00a6e848c43db884e9b89271b47a0" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42509 } }
I20260924 23:16:00.511404 2207 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5d00a6e848c43db884e9b89271b47a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d00a6e848c43db884e9b89271b47a0" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42509 } } }
I20260924 23:16:00.511408 2208 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5d00a6e848c43db884e9b89271b47a0. Latest consensus state: current_term: 1 leader_uuid: "b5d00a6e848c43db884e9b89271b47a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d00a6e848c43db884e9b89271b47a0" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42509 } } }
I20260924 23:16:00.511492 2207 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:00.511545 2208 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:00.511722 2215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:00.512368 2215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:00.514026 2215 catalog_manager.cc:1347] Generated new cluster ID: 33a080d2de7f41309a7b7ac70443f376
I20260924 23:16:00.514117 2215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:16:00 dist-test-slave-s1hw krb5kdc[2110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:00.521397 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:00.504518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.1.4.254:42509)
0924 23:16:00.504712 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:00.504722 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:00.504756 (+ 34us) server_negotiation.cc:407] Connection header received
0924 23:16:00.505341 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:00.505351 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:00.505427 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:00.505597 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:00.506326 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.507830 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.508710 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.509337 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.509479 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:00.517563 (+ 8084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:00.517569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:00.517570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:00.517585 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:00.519668 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:00.520101 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:00.520104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:00.520105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:00.520226 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:00.520422 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:00.520423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:00.520423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:00.520764 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:00.520929 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:00.521015 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 23:16:00.521161 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260924 23:16:00.522439 2215 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:00.523144 2215 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:00.536234 2215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0: Generated new TSK 0
I20260924 23:16:00.536458 2215 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:16:00.567307 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:42509
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:00.669634 2227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:00.669839 2227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:00.669862 2227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:00.671032 2227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:00.671074 2227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:00.671093 2227 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:16:00.671108 2227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:16:00.672610 2227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.4.254:42509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:00.672840 2227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:00.673048 2227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:00.678004 2233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:00.678027 2232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:00.677982 2235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:00.678205 2227 server_base.cc:1034] running on GCE node
I20260924 23:16:00.678382 2227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:00.678620 2227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:00.679757 2227 hybrid_clock.cc:648] HybridClock initialized: now 1790291760679742 us; error 25 us; skew 500 ppm
Sep 24 23:16:00 dist-test-slave-s1hw krb5kdc[2110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291760, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:00.682267 2227 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:16:00.682709 2227 webserver.cc:460] Webserver started at http://127.1.4.193:41657/ using document root <none> and password file <none>
I20260924 23:16:00.682924 2227 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:00.682971 2227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:00.683103 2227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:00.684162 2227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "33d239f6eb754359870bc71bb97f81c6"
format_stamp: "Formatted at 2026-09-24 23:16:00 on dist-test-slave-s1hw"
server_key: "4d735fedf264c61e0b7f6afb41544567"
server_key_iv: "a04bca35a6c55c5cedf5c84e1993c7c4"
server_key_version: "encryptionkey@0"
I20260924 23:16:00.684490 2227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "33d239f6eb754359870bc71bb97f81c6"
format_stamp: "Formatted at 2026-09-24 23:16:00 on dist-test-slave-s1hw"
server_key: "4d735fedf264c61e0b7f6afb41544567"
server_key_iv: "a04bca35a6c55c5cedf5c84e1993c7c4"
server_key_version: "encryptionkey@0"
I20260924 23:16:00.685681 2227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:16:00.686406 2242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:00.686558 2227 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:16:00.686638 2227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "33d239f6eb754359870bc71bb97f81c6"
format_stamp: "Formatted at 2026-09-24 23:16:00 on dist-test-slave-s1hw"
server_key: "4d735fedf264c61e0b7f6afb41544567"
server_key_iv: "a04bca35a6c55c5cedf5c84e1993c7c4"
server_key_version: "encryptionkey@0"
I20260924 23:16:00.686697 2227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:00.700591 2227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:00.702628 2227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:00.702766 2227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:00.703347 2227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:00.703421 2227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:00.703451 2227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:00.703473 2227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 23:16:00 dist-test-slave-s1hw krb5kdc[2110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291760, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:00.710111 2227 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:36807
I20260924 23:16:00.710170 2357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:36807 every 8 connection(s)
I20260924 23:16:00.710475 2227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:16:00.712590 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 2227
I20260924 23:16:00.712723 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:16:00.712888 1043 external_mini_cluster.cc:1428] Setting key 675975c7d84eec34215540d16b7e6f4d
I20260924 23:16:00.713212 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:00.703780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:33471 (local address 127.1.4.254:42509)
0924 23:16:00.703846 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:00.703848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:00.704125 (+ 277us) server_negotiation.cc:407] Connection header received
0924 23:16:00.704635 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:00.704636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:00.704649 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:00.704696 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:00.705356 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.705877 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.706928 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.707147 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.707205 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:00.710266 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:00.710267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:00.710267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:00.710276 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:00.711847 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:00.712217 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:00.712218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:00.712218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:00.712269 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:00.712444 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:00.712444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:00.712444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:00.712578 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:00.712646 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:00.712951 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 23:16:00.713123 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 23:16:00.713302 2251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:00.703772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42509 (local address 127.1.4.193:33471)
0924 23:16:00.703931 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:00.703973 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:00.704536 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:00.704781 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:00.704785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:00.705064 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:00.705260 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:00.705265 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.705964 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.705966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:00.706837 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:00.706840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.707221 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.707222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:00.707395 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:00.707950 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:00.707963 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:00.710175 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:00.711925 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:00.711931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:00.711933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:00.712159 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:00.712307 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:00.712308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:00.712308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:00.712400 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:00.712642 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:00.712648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:00.712825 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 23:16:00.712909 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 23:16:00.713077 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260924 23:16:00.713758 2358 heartbeater.cc:347] Connected to a master server at 127.1.4.254:42509
I20260924 23:16:00.713840 2358 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:00.714021 2358 heartbeater.cc:511] Master 127.1.4.254:42509 requested a full tablet report, sending...
I20260924 23:16:00.715246 2154 ts_manager.cc:194] Registered new tserver with Master: 33d239f6eb754359870bc71bb97f81c6 (127.1.4.193:36807)
I20260924 23:16:00.716163 2154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:33471
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
I20260924 23:16:00.720664 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:00.714679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42509 (local address 127.1.4.193:33911)
0924 23:16:00.715446 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:00.715457 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:00.715513 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:00.715668 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:00.715668 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:00.715839 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:00.715909 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:00.715911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.716737 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.716738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:00.717273 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:00.717275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.717606 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.717607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:00.717677 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:00.718079 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:00.718084 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:00.718359 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:00.719800 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:00.719801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:00.719801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:00.719958 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:00.720118 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:00.720118 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:00.720119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:00.720174 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:00.720372 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:00.720373 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:00.720414 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 23:16:00.720502 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 23:16:00.720591 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":62,"threads_started":1}
I20260924 23:16:00.720714 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:00.714735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:33911 (local address 127.1.4.254:42509)
0924 23:16:00.714792 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:00.714794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:00.715587 (+ 793us) server_negotiation.cc:407] Connection header received
0924 23:16:00.715608 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:00.715609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:00.715618 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:00.715657 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:00.715992 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.716651 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.717346 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.717549 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.717601 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:00.718435 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:00.718436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:00.718436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:00.718443 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:00.719730 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:00.720015 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:00.720016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:00.720016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:00.720068 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:00.720221 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:00.720222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:00.720223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:00.720324 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:00.720376 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:00.720557 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 23:16:00.720633 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:16:00.738763 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:42509
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:00.820142 2367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:00.820353 2367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:00.820376 2367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:00.821537 2367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:00.821573 2367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:00.821592 2367 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:16:00.821609 2367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:16:00.823338 2367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.4.254:42509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:00.823596 2367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:00.823783 2367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:00.828683 2372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:00.828735 2373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:00.828738 2375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:00.828785 2367 server_base.cc:1034] running on GCE node
I20260924 23:16:00.829066 2367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:00.829298 2367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:00.830471 2367 hybrid_clock.cc:648] HybridClock initialized: now 1790291760830454 us; error 32 us; skew 500 ppm
Sep 24 23:16:00 dist-test-slave-s1hw krb5kdc[2110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291760, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:00.833077 2367 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:16:00.833520 2367 webserver.cc:460] Webserver started at http://127.1.4.194:44075/ using document root <none> and password file <none>
I20260924 23:16:00.833719 2367 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:00.833762 2367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:00.833896 2367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:00.834965 2367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "5cb0e3c157e84ffbb575f28ba8272001"
format_stamp: "Formatted at 2026-09-24 23:16:00 on dist-test-slave-s1hw"
server_key: "5998501a2b3174a8ef27c4b78326a16f"
server_key_iv: "042869145f151058034bdb98d473d913"
server_key_version: "encryptionkey@0"
I20260924 23:16:00.835289 2367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "5cb0e3c157e84ffbb575f28ba8272001"
format_stamp: "Formatted at 2026-09-24 23:16:00 on dist-test-slave-s1hw"
server_key: "5998501a2b3174a8ef27c4b78326a16f"
server_key_iv: "042869145f151058034bdb98d473d913"
server_key_version: "encryptionkey@0"
I20260924 23:16:00.836611 2367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:16:00.837401 2382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:00.837579 2367 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:16:00.837668 2367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "5cb0e3c157e84ffbb575f28ba8272001"
format_stamp: "Formatted at 2026-09-24 23:16:00 on dist-test-slave-s1hw"
server_key: "5998501a2b3174a8ef27c4b78326a16f"
server_key_iv: "042869145f151058034bdb98d473d913"
server_key_version: "encryptionkey@0"
I20260924 23:16:00.837735 2367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:00.864683 2367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:00.866984 2367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:00.867106 2367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:00.867762 2367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:00.867815 2367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:00.867844 2367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:00.867861 2367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 23:16:00 dist-test-slave-s1hw krb5kdc[2110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291760, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:00.874683 2367 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:42275
I20260924 23:16:00.875052 2367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:16:00.875032 2497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:42275 every 8 connection(s)
I20260924 23:16:00.877063 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:00.868205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:41619 (local address 127.1.4.254:42509)
0924 23:16:00.868256 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:00.868258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:00.868490 (+ 232us) server_negotiation.cc:407] Connection header received
0924 23:16:00.868967 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:00.868968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:00.868982 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:00.869057 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:00.869692 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.870192 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.871235 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.871578 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.871642 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:00.874272 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:00.874273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:00.874274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:00.874281 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:00.875740 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:00.876240 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:00.876242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:00.876242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:00.876302 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:00.876526 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:00.876528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:00.876528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:00.876657 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:00.876713 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:00.876936 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 23:16:00.876997 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:16:00.877292 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:00.868193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42509 (local address 127.1.4.194:41619)
0924 23:16:00.868358 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:00.868401 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:00.868876 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:00.869121 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:00.869125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:00.869402 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:00.869598 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:00.869604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.870276 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.870277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:00.871149 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:00.871152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.871693 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.871694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:00.871863 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:00.872420 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:00.872431 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:00.874178 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:00.875812 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:00.875815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:00.875816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:00.876167 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:00.876360 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:00.876360 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:00.876361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:00.876452 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:00.876715 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:00.876719 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:00.876834 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 23:16:00.876916 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 23:16:00.877103 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260924 23:16:00.877761 2498 heartbeater.cc:347] Connected to a master server at 127.1.4.254:42509
I20260924 23:16:00.877844 2498 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:00.878012 2498 heartbeater.cc:511] Master 127.1.4.254:42509 requested a full tablet report, sending...
I20260924 23:16:00.878381 2154 ts_manager.cc:194] Registered new tserver with Master: 5cb0e3c157e84ffbb575f28ba8272001 (127.1.4.194:42275)
I20260924 23:16:00.878867 2154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:41619
I20260924 23:16:00.883630 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 2367
I20260924 23:16:00.883725 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:16:00.883885 1043 external_mini_cluster.cc:1428] Setting key 73b27a30011b5e82c50dee9da90c8b45
I20260924 23:16:00.883934 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:00.878810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:37177 (local address 127.1.4.254:42509)
0924 23:16:00.878848 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:00.878850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:00.878975 (+ 125us) server_negotiation.cc:407] Connection header received
0924 23:16:00.879027 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:00.879027 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:00.879038 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:00.879091 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:00.879439 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.879919 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.880618 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.880814 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.880867 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:00.881692 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:00.881693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:00.881694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:00.881700 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:00.882830 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:00.883112 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:00.883113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:00.883114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:00.883168 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:00.883325 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:00.883325 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:00.883326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:00.883543 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:00.883610 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:00.883773 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 23:16:00.883848 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:16:00.884028 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:00.878761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42509 (local address 127.1.4.194:37177)
0924 23:16:00.878904 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:00.878915 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:00.878980 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:00.879134 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:00.879135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:00.879290 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:00.879348 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:00.879349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.880021 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.880021 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:00.880546 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:00.880548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:00.880871 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:00.880872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:00.880945 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:00.881333 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:00.881337 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:00.881617 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:00.882898 (+ 1281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:00.882899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:00.882900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:00.883055 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:00.883220 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:00.883220 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:00.883221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:00.883275 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:00.883625 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:00.883626 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:00.883668 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 23:16:00.883733 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 23:16:00.883837 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:16:00.909426 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:42509
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:00.988742 2507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:00.988955 2507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:00.988987 2507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:00.990242 2507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:00.990294 2507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:00.990321 2507 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:16:00.990339 2507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:16:00.991828 2507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.4.254:42509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:00.992076 2507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:00.992297 2507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:00.997167 2515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:00.997174 2512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:00.997174 2507 server_base.cc:1034] running on GCE node
W20260924 23:16:00.997400 2513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:00.997638 2507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:00.997890 2507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:00.999023 2507 hybrid_clock.cc:648] HybridClock initialized: now 1790291760999012 us; error 32 us; skew 500 ppm
Sep 24 23:16:00 dist-test-slave-s1hw krb5kdc[2110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291760, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:01.001572 2507 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:16:01.002012 2507 webserver.cc:460] Webserver started at http://127.1.4.195:37747/ using document root <none> and password file <none>
I20260924 23:16:01.002207 2507 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:01.002246 2507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:01.002339 2507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:01.003333 2507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "78cb8af4b9e344da8bee15a8566028e7"
format_stamp: "Formatted at 2026-09-24 23:16:00 on dist-test-slave-s1hw"
server_key: "48a670dece6a301058618bd67fd064fd"
server_key_iv: "f21004eb217465984e4b16354d2a2a23"
server_key_version: "encryptionkey@0"
I20260924 23:16:01.003739 2507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "78cb8af4b9e344da8bee15a8566028e7"
format_stamp: "Formatted at 2026-09-24 23:16:00 on dist-test-slave-s1hw"
server_key: "48a670dece6a301058618bd67fd064fd"
server_key_iv: "f21004eb217465984e4b16354d2a2a23"
server_key_version: "encryptionkey@0"
I20260924 23:16:01.005102 2507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:01.005766 2522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:01.005998 2507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:16:01.006065 2507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "78cb8af4b9e344da8bee15a8566028e7"
format_stamp: "Formatted at 2026-09-24 23:16:00 on dist-test-slave-s1hw"
server_key: "48a670dece6a301058618bd67fd064fd"
server_key_iv: "f21004eb217465984e4b16354d2a2a23"
server_key_version: "encryptionkey@0"
I20260924 23:16:01.006129 2507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:01.033039 2507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:01.035238 2507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:01.035348 2507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:01.035854 2507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:01.035897 2507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:01.035929 2507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:01.035953 2507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 23:16:01 dist-test-slave-s1hw krb5kdc[2110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291760, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:01.042528 2507 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:38981
I20260924 23:16:01.042608 2637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:38981 every 8 connection(s)
I20260924 23:16:01.042900 2507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:16:01.044118 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 2507
I20260924 23:16:01.044217 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:16:01.044379 1043 external_mini_cluster.cc:1428] Setting key 628c5af4e4401a3a724ba1fc55fa4ed7
I20260924 23:16:01.046065 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.036329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:49545 (local address 127.1.4.254:42509)
0924 23:16:01.036371 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.036373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.036650 (+ 277us) server_negotiation.cc:407] Connection header received
0924 23:16:01.037202 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.037203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.037215 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.037267 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:01.038063 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.038819 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.040166 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.040376 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.040440 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.043062 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:01.043064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:01.043064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:01.043072 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:01.044699 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:01.045168 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:01.045169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:01.045170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:01.045228 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:01.045443 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:01.045451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:01.045451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:01.045570 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:01.045624 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.045897 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.045976 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:16:01.046250 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.036342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42509 (local address 127.1.4.195:49545)
0924 23:16:01.036528 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:01.036565 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:01.037063 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:01.037338 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:01.037342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:01.037621 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:01.037897 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.037905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.038991 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.038992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.040089 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.040091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.040465 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.040466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.040627 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.041169 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:01.041176 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:01.042920 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:01.044798 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:01.044804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:01.044807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:01.045095 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:01.045287 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:01.045288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:01.045288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:01.045389 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:01.045624 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:01.045629 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:01.045777 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 23:16:01.045869 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 23:16:01.046037 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260924 23:16:01.046721 2638 heartbeater.cc:347] Connected to a master server at 127.1.4.254:42509
I20260924 23:16:01.046804 2638 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:01.046991 2638 heartbeater.cc:511] Master 127.1.4.254:42509 requested a full tablet report, sending...
I20260924 23:16:01.047345 2154 ts_manager.cc:194] Registered new tserver with Master: 78cb8af4b9e344da8bee15a8566028e7 (127.1.4.195:38981)
I20260924 23:16:01.047892 2154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:49545
I20260924 23:16:01.052796 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.047678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:57795 (local address 127.1.4.254:42509)
0924 23:16:01.048003 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.048005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.048012 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:16:01.048029 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.048029 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.048038 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.048078 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:01.048392 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.048855 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.049509 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.049705 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.049756 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.050566 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:01.050567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:01.050568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:01.050574 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:01.051810 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:01.052110 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:01.052111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:01.052111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:01.052161 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:01.052327 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:01.052328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:01.052328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:01.052430 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:01.052481 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.052634 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.052703 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308}
I20260924 23:16:01.052809 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.047598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42509 (local address 127.1.4.195:57795)
0924 23:16:01.047716 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:01.047727 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:01.047775 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:01.048095 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:01.048096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:01.048261 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:01.048327 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.048329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.048936 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.048937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.049440 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.049441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.049759 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.049760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.049831 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.050219 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:01.050223 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:01.050494 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:01.051885 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:01.051886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:01.051887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:01.052050 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:01.052220 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:01.052221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:01.052221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:01.052279 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:01.052479 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:01.052480 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:01.052534 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 23:16:01.052603 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 23:16:01.052704 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
I20260924 23:16:01.056100 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 23:16:01 dist-test-slave-s1hw krb5kdc[2110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291761, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 23:16:01 dist-test-slave-s1hw krb5kdc[2110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291761, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:01.086442 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.079210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53922 (local address 127.1.4.254:42509)
0924 23:16:01.079301 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.079303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.079314 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:16:01.079345 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.079347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.079358 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.079465 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:01.080101 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.080661 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.081565 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.081754 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.081811 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.084171 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:01.084172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:01.084173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:01.084182 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:01.085365 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:01.085743 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:01.085744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:01.085745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:01.085796 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:01.085963 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:01.085964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:01.085964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:01.086077 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:01.086170 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.086269 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.086347 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 23:16:01.088223 2154 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:16:01.089860 2154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:01.096014 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.092317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:36807 (local address 127.0.0.1:60076)
0924 23:16:01.092548 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:01.092591 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:01.092657 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:01.092879 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:01.092881 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:01.092883 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:01.092963 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.092964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.093942 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.093943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.094792 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.094794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.095686 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.095686 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.095769 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.095782 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 23:16:01.095857 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 23:16:01.095912 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":94,"threads_started":1}
I20260924 23:16:01.096089 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.092570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.1.4.193:36807)
0924 23:16:01.092728 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.092732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.092744 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:16:01.092784 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.092786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.092801 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.092878 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:01.093106 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.093865 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.094877 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.095532 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.095595 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.095655 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.095888 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.095926 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
I20260924 23:16:01.096566 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.092318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:38981 (local address 127.0.0.1:47996)
0924 23:16:01.092715 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:01.092725 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:01.092789 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:01.092972 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:01.092972 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:01.092975 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:01.093030 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.093032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.094321 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.094322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.095501 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.095503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.096346 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.096347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.096407 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.096410 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:01.096459 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 23:16:01.096497 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":100,"threads_started":1}
I20260924 23:16:01.096912 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.092406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47996 (local address 127.1.4.195:38981)
0924 23:16:01.092577 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.092583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.092809 (+ 226us) server_negotiation.cc:407] Connection header received
0924 23:16:01.092854 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.092857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.092872 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.092950 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:01.093114 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.094228 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.095592 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.096274 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.096333 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.096387 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.096509 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.096552 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
I20260924 23:16:01.096793 2292 tablet_service.cc:1467] Processing CreateTablet for tablet 5fdf3ffd3045446db8866d7f93b6e90b (DEFAULT_TABLE table=test-table [id=10911d2704d04ad08125ce76211ade05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:01.097182 2292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fdf3ffd3045446db8866d7f93b6e90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:01.097393 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.092317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:42275 (local address 127.0.0.1:46354)
0924 23:16:01.092751 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:01.092759 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:01.092799 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:01.093194 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:01.093195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:01.093197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:01.093241 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.093242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.094410 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.094411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.095505 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.095507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.097155 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.097155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.097242 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.097247 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:16:01.097289 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 23:16:01.097327 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":3,"thread_start_us":40,"threads_started":1}
I20260924 23:16:01.097499 2572 tablet_service.cc:1467] Processing CreateTablet for tablet 5fdf3ffd3045446db8866d7f93b6e90b (DEFAULT_TABLE table=test-table [id=10911d2704d04ad08125ce76211ade05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:01.097635 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.092541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.1.4.194:42275)
0924 23:16:01.092694 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.092698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.093013 (+ 315us) server_negotiation.cc:407] Connection header received
0924 23:16:01.093052 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.093054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.093068 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.093134 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:01.093307 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.094335 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.096423 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.097001 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.097069 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.097122 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.097481 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.097538 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260924 23:16:01.097816 2572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fdf3ffd3045446db8866d7f93b6e90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:01.098177 2432 tablet_service.cc:1467] Processing CreateTablet for tablet 5fdf3ffd3045446db8866d7f93b6e90b (DEFAULT_TABLE table=test-table [id=10911d2704d04ad08125ce76211ade05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:01.098440 2432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fdf3ffd3045446db8866d7f93b6e90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:01.100914 2661 tablet_bootstrap.cc:492] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7: Bootstrap starting.
I20260924 23:16:01.101254 2662 tablet_bootstrap.cc:492] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6: Bootstrap starting.
I20260924 23:16:01.101276 2663 tablet_bootstrap.cc:492] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001: Bootstrap starting.
I20260924 23:16:01.101881 2663 tablet_bootstrap.cc:654] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:01.101881 2662 tablet_bootstrap.cc:654] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:01.101889 2661 tablet_bootstrap.cc:654] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:01.102180 2661 log.cc:824] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:01.102216 2663 log.cc:824] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:01.102288 2662 log.cc:824] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:01.102917 2663 tablet_bootstrap.cc:492] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001: No bootstrap required, opened a new log
I20260924 23:16:01.102944 2661 tablet_bootstrap.cc:492] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7: No bootstrap required, opened a new log
I20260924 23:16:01.102977 2663 ts_tablet_manager.cc:1397] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:01.103000 2661 ts_tablet_manager.cc:1397] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:01.104525 2663 raft_consensus.cc:348] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } }
I20260924 23:16:01.104549 2661 raft_consensus.cc:348] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } }
I20260924 23:16:01.104667 2663 raft_consensus.cc:374] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:01.104667 2661 raft_consensus.cc:374] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:01.104688 2663 raft_consensus.cc:729] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb0e3c157e84ffbb575f28ba8272001, State: Initialized, Role: FOLLOWER
I20260924 23:16:01.104697 2661 raft_consensus.cc:729] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78cb8af4b9e344da8bee15a8566028e7, State: Initialized, Role: FOLLOWER
I20260924 23:16:01.104792 2663 consensus_queue.cc:260] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } }
I20260924 23:16:01.104799 2661 consensus_queue.cc:260] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } }
I20260924 23:16:01.105057 2663 ts_tablet_manager.cc:1428] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:16:01.105058 2661 ts_tablet_manager.cc:1428] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:16:01.105099 2498 heartbeater.cc:503] Master 127.1.4.254:42509 was elected leader, sending a full tablet report...
I20260924 23:16:01.105182 2638 heartbeater.cc:503] Master 127.1.4.254:42509 was elected leader, sending a full tablet report...
I20260924 23:16:01.105515 2662 tablet_bootstrap.cc:492] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6: No bootstrap required, opened a new log
I20260924 23:16:01.105583 2662 ts_tablet_manager.cc:1397] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 23:16:01.107131 2662 raft_consensus.cc:348] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } }
I20260924 23:16:01.107257 2662 raft_consensus.cc:374] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:01.107277 2662 raft_consensus.cc:729] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d239f6eb754359870bc71bb97f81c6, State: Initialized, Role: FOLLOWER
I20260924 23:16:01.107360 2662 consensus_queue.cc:260] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } }
I20260924 23:16:01.107633 2662 ts_tablet_manager.cc:1428] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:16:01.107679 2358 heartbeater.cc:503] Master 127.1.4.254:42509 was elected leader, sending a full tablet report...
W20260924 23:16:01.125670 2499 tablet.cc:2369] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:16:01.180562 2668 raft_consensus.cc:482] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:01.180702 2668 raft_consensus.cc:504] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } }
I20260924 23:16:01.181102 2668 leader_election.cc:288] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33d239f6eb754359870bc71bb97f81c6 (127.1.4.193:36807), 5cb0e3c157e84ffbb575f28ba8272001 (127.1.4.194:42275)
I20260924 23:16:01.184017 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.181377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:42771 (local address 127.1.4.193:36807)
0924 23:16:01.181423 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.181425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.181461 (+ 36us) server_negotiation.cc:407] Connection header received
0924 23:16:01.181545 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.181546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.181560 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.181626 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:01.181773 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.182355 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.183315 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.183690 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.183742 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.183752 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.183917 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.183953 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:16:01.184024 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.181382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:41303 (local address 127.1.4.194:42275)
0924 23:16:01.181452 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.181453 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.181666 (+ 213us) server_negotiation.cc:407] Connection header received
0924 23:16:01.181693 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.181695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.181707 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.181757 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:01.181971 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.182430 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.183314 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.183713 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.183779 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.183790 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.183927 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.183965 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 23:16:01.184024 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.181322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:36807 (local address 127.1.4.195:42771)
0924 23:16:01.181400 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:01.181413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:01.181471 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:01.181638 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:01.181639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:01.181641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:01.181710 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.181713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.182458 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.182459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.183225 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.183227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.183762 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.183763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.183845 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.183848 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:01.183891 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 23:16:01.183936 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 23:16:01.184038 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.181326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:42275 (local address 127.1.4.195:41303)
0924 23:16:01.181562 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:01.181572 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:01.181629 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:01.181765 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:01.181766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:01.181767 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:01.181836 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.181837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.182512 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.182513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.183227 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.183228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.183791 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.183792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.183855 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.183858 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:01.183908 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 23:16:01.183949 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":56,"thread_start_us":110,"threads_started":1}
I20260924 23:16:01.184378 2452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdf3ffd3045446db8866d7f93b6e90b" candidate_uuid: "78cb8af4b9e344da8bee15a8566028e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb0e3c157e84ffbb575f28ba8272001" is_pre_election: true
I20260924 23:16:01.184489 2312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdf3ffd3045446db8866d7f93b6e90b" candidate_uuid: "78cb8af4b9e344da8bee15a8566028e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d239f6eb754359870bc71bb97f81c6" is_pre_election: true
I20260924 23:16:01.184538 2452 raft_consensus.cc:2454] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78cb8af4b9e344da8bee15a8566028e7 in term 0.
I20260924 23:16:01.184619 2312 raft_consensus.cc:2454] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78cb8af4b9e344da8bee15a8566028e7 in term 0.
I20260924 23:16:01.184782 2524 leader_election.cc:302] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb0e3c157e84ffbb575f28ba8272001, 78cb8af4b9e344da8bee15a8566028e7; no voters:
I20260924 23:16:01.184926 2668 raft_consensus.cc:2781] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:01.184993 2668 raft_consensus.cc:482] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:01.185010 2668 raft_consensus.cc:3037] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:01.185698 2668 raft_consensus.cc:504] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } }
I20260924 23:16:01.185817 2668 leader_election.cc:288] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [CANDIDATE]: Term 1 election: Requested vote from peers 33d239f6eb754359870bc71bb97f81c6 (127.1.4.193:36807), 5cb0e3c157e84ffbb575f28ba8272001 (127.1.4.194:42275)
I20260924 23:16:01.185982 2312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdf3ffd3045446db8866d7f93b6e90b" candidate_uuid: "78cb8af4b9e344da8bee15a8566028e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d239f6eb754359870bc71bb97f81c6"
I20260924 23:16:01.186074 2312 raft_consensus.cc:3037] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:01.186081 2452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdf3ffd3045446db8866d7f93b6e90b" candidate_uuid: "78cb8af4b9e344da8bee15a8566028e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb0e3c157e84ffbb575f28ba8272001"
I20260924 23:16:01.186151 2452 raft_consensus.cc:3037] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:01.186955 2452 raft_consensus.cc:2454] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78cb8af4b9e344da8bee15a8566028e7 in term 1.
I20260924 23:16:01.186955 2312 raft_consensus.cc:2454] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78cb8af4b9e344da8bee15a8566028e7 in term 1.
I20260924 23:16:01.187120 2524 leader_election.cc:302] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb0e3c157e84ffbb575f28ba8272001, 78cb8af4b9e344da8bee15a8566028e7; no voters:
I20260924 23:16:01.187227 2668 raft_consensus.cc:2781] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:01.187417 2668 raft_consensus.cc:686] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [term 1 LEADER]: Becoming Leader. State: Replica: 78cb8af4b9e344da8bee15a8566028e7, State: Running, Role: LEADER
I20260924 23:16:01.187516 2668 consensus_queue.cc:237] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } }
I20260924 23:16:01.188308 2154 catalog_manager.cc:5495] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78cb8af4b9e344da8bee15a8566028e7 (127.1.4.195). New cstate: current_term: 1 leader_uuid: "78cb8af4b9e344da8bee15a8566028e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } health_report { overall_health: UNKNOWN } } }
W20260924 23:16:01.211187 2359 tablet.cc:2369] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:16:01.213632 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.210857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.1.4.195:38981)
0924 23:16:01.210913 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.210916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.210948 (+ 32us) server_negotiation.cc:407] Connection header received
0924 23:16:01.211103 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.211104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.211124 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.211179 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:01.211314 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.211874 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.212739 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.212928 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.212984 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.213185 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:01.213285 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:01.213385 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.213509 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.213533 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 23:16:01.215399 2452 raft_consensus.cc:1261] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001 [term 1 FOLLOWER]: Refusing update from remote peer 78cb8af4b9e344da8bee15a8566028e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:01.215399 2312 raft_consensus.cc:1261] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6 [term 1 FOLLOWER]: Refusing update from remote peer 78cb8af4b9e344da8bee15a8566028e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:01.215718 2668 consensus_queue.cc:1035] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:01.215842 2671 consensus_queue.cc:1035] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:01.218001 2674 mvcc.cc:205] Tried to move back new op lower bound from 7333035053935669248 to 7333035053824696320. Current Snapshot: MvccSnapshot[applied={T|T < 7333035053935669248}]
I20260924 23:16:01.218344 2675 mvcc.cc:205] Tried to move back new op lower bound from 7333035053935669248 to 7333035053824696320. Current Snapshot: MvccSnapshot[applied={T|T < 7333035053935669248}]
I20260924 23:16:01.221457 2154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:01.221546 2154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:01.222259 2154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0: Sending DeleteTablet for 3 replicas of tablet 5fdf3ffd3045446db8866d7f93b6e90b
I20260924 23:16:01.222561 2572 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fdf3ffd3045446db8866d7f93b6e90b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:01 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:47996
I20260924 23:16:01.222577 2432 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fdf3ffd3045446db8866d7f93b6e90b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:01 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:46354
I20260924 23:16:01.222644 2292 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fdf3ffd3045446db8866d7f93b6e90b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:01 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:60076
I20260924 23:16:01.222766 2685 tablet_replica.cc:330] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001: stopping tablet replica
I20260924 23:16:01.222859 2685 raft_consensus.cc:2229] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:01.222965 2685 raft_consensus.cc:2258] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:01.223078 2684 tablet_replica.cc:330] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7: stopping tablet replica
I20260924 23:16:01.223166 2684 raft_consensus.cc:2229] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:16:01.223151 2154 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 23:16:01.223280 2684 raft_consensus.cc:2258] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 23:16:01.223282 2154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:01.223749 2684 ts_tablet_manager.cc:1905] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:01.223788 2685 ts_tablet_manager.cc:1905] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:01.224573 2432 tablet_service.cc:1467] Processing CreateTablet for tablet dfa4013f8f934ecf8475ccc0cb9923f4 (DEFAULT_TABLE table=test-table [id=a0bac1399a914500a962262333e58b5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:01.224684 2432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa4013f8f934ecf8475ccc0cb9923f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:01.225592 2684 ts_tablet_manager.cc:1918] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:01.225654 2684 log.cc:1196] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/5fdf3ffd3045446db8866d7f93b6e90b
I20260924 23:16:01.225754 2685 ts_tablet_manager.cc:1918] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:01.225807 2685 log.cc:1196] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/5fdf3ffd3045446db8866d7f93b6e90b
I20260924 23:16:01.225905 2684 ts_tablet_manager.cc:1939] T 5fdf3ffd3045446db8866d7f93b6e90b P 78cb8af4b9e344da8bee15a8566028e7: Deleting consensus metadata
I20260924 23:16:01.225972 2572 tablet_service.cc:1467] Processing CreateTablet for tablet dfa4013f8f934ecf8475ccc0cb9923f4 (DEFAULT_TABLE table=test-table [id=a0bac1399a914500a962262333e58b5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:01.226011 2685 ts_tablet_manager.cc:1939] T 5fdf3ffd3045446db8866d7f93b6e90b P 5cb0e3c157e84ffbb575f28ba8272001: Deleting consensus metadata
I20260924 23:16:01.226086 2572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa4013f8f934ecf8475ccc0cb9923f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:01.226161 2686 tablet_replica.cc:330] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6: stopping tablet replica
I20260924 23:16:01.226264 2686 raft_consensus.cc:2229] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:01.226357 2686 raft_consensus.cc:2258] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:01.226497 2141 catalog_manager.cc:4841] TS 78cb8af4b9e344da8bee15a8566028e7 (127.1.4.195:38981): tablet 5fdf3ffd3045446db8866d7f93b6e90b (table test-table [id=10911d2704d04ad08125ce76211ade05]) successfully deleted
I20260924 23:16:01.226835 2686 ts_tablet_manager.cc:1905] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:01.227475 2663 tablet_bootstrap.cc:492] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001: Bootstrap starting.
I20260924 23:16:01.227802 2661 tablet_bootstrap.cc:492] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7: Bootstrap starting.
I20260924 23:16:01.227844 2663 tablet_bootstrap.cc:654] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:01.228117 2686 ts_tablet_manager.cc:1918] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:01.228160 2661 tablet_bootstrap.cc:654] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:01.228185 2686 log.cc:1196] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/wal/wals/5fdf3ffd3045446db8866d7f93b6e90b
I20260924 23:16:01.228271 2140 catalog_manager.cc:4841] TS 5cb0e3c157e84ffbb575f28ba8272001 (127.1.4.194:42275): tablet 5fdf3ffd3045446db8866d7f93b6e90b (table test-table [id=10911d2704d04ad08125ce76211ade05]) successfully deleted
I20260924 23:16:01.228415 2686 ts_tablet_manager.cc:1939] T 5fdf3ffd3045446db8866d7f93b6e90b P 33d239f6eb754359870bc71bb97f81c6: Deleting consensus metadata
I20260924 23:16:01.228600 2292 tablet_service.cc:1467] Processing CreateTablet for tablet dfa4013f8f934ecf8475ccc0cb9923f4 (DEFAULT_TABLE table=test-table [id=a0bac1399a914500a962262333e58b5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:01.228691 2292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa4013f8f934ecf8475ccc0cb9923f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:01.228857 2661 tablet_bootstrap.cc:492] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7: No bootstrap required, opened a new log
I20260924 23:16:01.228897 2661 ts_tablet_manager.cc:1397] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:01.228928 2139 catalog_manager.cc:4841] TS 33d239f6eb754359870bc71bb97f81c6 (127.1.4.193:36807): tablet 5fdf3ffd3045446db8866d7f93b6e90b (table test-table [id=10911d2704d04ad08125ce76211ade05]) successfully deleted
I20260924 23:16:01.229023 2661 raft_consensus.cc:348] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.229094 2661 raft_consensus.cc:374] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:01.229113 2661 raft_consensus.cc:729] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78cb8af4b9e344da8bee15a8566028e7, State: Initialized, Role: FOLLOWER
I20260924 23:16:01.229152 2663 tablet_bootstrap.cc:492] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001: No bootstrap required, opened a new log
I20260924 23:16:01.229152 2661 consensus_queue.cc:260] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.229190 2663 ts_tablet_manager.cc:1397] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:01.229231 2661 ts_tablet_manager.cc:1428] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:01.229321 2663 raft_consensus.cc:348] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.229394 2663 raft_consensus.cc:374] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:01.229415 2663 raft_consensus.cc:729] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb0e3c157e84ffbb575f28ba8272001, State: Initialized, Role: FOLLOWER
I20260924 23:16:01.229453 2663 consensus_queue.cc:260] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.229527 2663 ts_tablet_manager.cc:1428] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:01.229781 2662 tablet_bootstrap.cc:492] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6: Bootstrap starting.
I20260924 23:16:01.230118 2662 tablet_bootstrap.cc:654] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:01.230682 2662 tablet_bootstrap.cc:492] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6: No bootstrap required, opened a new log
I20260924 23:16:01.230727 2662 ts_tablet_manager.cc:1397] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:01.231011 2662 raft_consensus.cc:348] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.231069 2662 raft_consensus.cc:374] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:01.231086 2662 raft_consensus.cc:729] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d239f6eb754359870bc71bb97f81c6, State: Initialized, Role: FOLLOWER
I20260924 23:16:01.231127 2662 consensus_queue.cc:260] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.231221 2662 ts_tablet_manager.cc:1428] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:01.235451 2679 raft_consensus.cc:482] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:01.235513 2679 raft_consensus.cc:504] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.235664 2679 leader_election.cc:288] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33d239f6eb754359870bc71bb97f81c6 (127.1.4.193:36807), 5cb0e3c157e84ffbb575f28ba8272001 (127.1.4.194:42275)
I20260924 23:16:01.235793 2312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa4013f8f934ecf8475ccc0cb9923f4" candidate_uuid: "78cb8af4b9e344da8bee15a8566028e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d239f6eb754359870bc71bb97f81c6" is_pre_election: true
I20260924 23:16:01.235807 2452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa4013f8f934ecf8475ccc0cb9923f4" candidate_uuid: "78cb8af4b9e344da8bee15a8566028e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb0e3c157e84ffbb575f28ba8272001" is_pre_election: true
I20260924 23:16:01.235864 2312 raft_consensus.cc:2454] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78cb8af4b9e344da8bee15a8566028e7 in term 0.
I20260924 23:16:01.235889 2452 raft_consensus.cc:2454] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78cb8af4b9e344da8bee15a8566028e7 in term 0.
I20260924 23:16:01.235999 2523 leader_election.cc:302] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d239f6eb754359870bc71bb97f81c6, 78cb8af4b9e344da8bee15a8566028e7; no voters:
I20260924 23:16:01.236085 2679 raft_consensus.cc:2781] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:01.236114 2679 raft_consensus.cc:482] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:01.236137 2679 raft_consensus.cc:3037] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:01.236835 2679 raft_consensus.cc:504] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.236944 2679 leader_election.cc:288] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [CANDIDATE]: Term 1 election: Requested vote from peers 33d239f6eb754359870bc71bb97f81c6 (127.1.4.193:36807), 5cb0e3c157e84ffbb575f28ba8272001 (127.1.4.194:42275)
I20260924 23:16:01.237071 2312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa4013f8f934ecf8475ccc0cb9923f4" candidate_uuid: "78cb8af4b9e344da8bee15a8566028e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d239f6eb754359870bc71bb97f81c6"
I20260924 23:16:01.237144 2312 raft_consensus.cc:3037] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:01.237807 2312 raft_consensus.cc:2454] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78cb8af4b9e344da8bee15a8566028e7 in term 1.
I20260924 23:16:01.237962 2523 leader_election.cc:302] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d239f6eb754359870bc71bb97f81c6, 78cb8af4b9e344da8bee15a8566028e7; no voters:
I20260924 23:16:01.237977 2452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa4013f8f934ecf8475ccc0cb9923f4" candidate_uuid: "78cb8af4b9e344da8bee15a8566028e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb0e3c157e84ffbb575f28ba8272001"
I20260924 23:16:01.238042 2452 raft_consensus.cc:3037] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:01.238104 2679 raft_consensus.cc:2781] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:01.238149 2679 raft_consensus.cc:686] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [term 1 LEADER]: Becoming Leader. State: Replica: 78cb8af4b9e344da8bee15a8566028e7, State: Running, Role: LEADER
I20260924 23:16:01.238196 2679 consensus_queue.cc:237] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.238746 2452 raft_consensus.cc:2454] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78cb8af4b9e344da8bee15a8566028e7 in term 1.
I20260924 23:16:01.239181 2154 catalog_manager.cc:5495] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78cb8af4b9e344da8bee15a8566028e7 (127.1.4.195). New cstate: current_term: 1 leader_uuid: "78cb8af4b9e344da8bee15a8566028e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } health_report { overall_health: HEALTHY } } }
Sep 24 23:16:01 dist-test-slave-s1hw krb5kdc[2110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291760, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:01.271216 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.264248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53932 (local address 127.1.4.254:42509)
0924 23:16:01.264312 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.264314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.264458 (+ 144us) server_negotiation.cc:407] Connection header received
0924 23:16:01.264536 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.264537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.264549 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.264600 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:01.265095 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.265600 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.266312 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.266496 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.266557 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.268666 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:01.268668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:01.268668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:01.268675 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:01.270093 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:01.270439 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:01.270440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:01.270441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:01.270495 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:01.270650 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:01.270651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:01.270651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:01.270775 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:01.270829 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.270966 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.271130 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":39}
I20260924 23:16:01.271219 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.264167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42509 (local address 127.0.0.1:53932)
0924 23:16:01.264345 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:01.264356 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:01.264469 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:01.264621 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:01.264623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:01.264958 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:01.265013 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.265016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.265678 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.265679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.266229 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.266231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.266560 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.266561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.266606 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.267155 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:01.267160 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:01.268576 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:01.270164 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:01.270172 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:01.270175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:01.270378 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:01.270545 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:01.270546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:01.270546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:01.270597 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:01.270825 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:01.270833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:01.270881 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 23:16:01.270938 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 23:16:01.271074 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":4,"thread_start_us":96,"threads_started":1}
I20260924 23:16:01.273563 2154 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:53932:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 23:16:01.273748 2154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:01.275025 2292 tablet_service.cc:1467] Processing CreateTablet for tablet 5bab86a884ca408f9ce5d0022076ed27 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c9168547ec5451fbb622e4a652517e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:16:01.275175 2292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bab86a884ca408f9ce5d0022076ed27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:01.275496 2432 tablet_service.cc:1467] Processing CreateTablet for tablet 5bab86a884ca408f9ce5d0022076ed27 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c9168547ec5451fbb622e4a652517e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:16:01.275656 2432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bab86a884ca408f9ce5d0022076ed27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:01.276674 2663 tablet_bootstrap.cc:492] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001: Bootstrap starting.
I20260924 23:16:01.276773 2662 tablet_bootstrap.cc:492] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6: Bootstrap starting.
I20260924 23:16:01.276751 2572 tablet_service.cc:1467] Processing CreateTablet for tablet 5bab86a884ca408f9ce5d0022076ed27 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c9168547ec5451fbb622e4a652517e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:16:01.276870 2572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bab86a884ca408f9ce5d0022076ed27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:01.277086 2663 tablet_bootstrap.cc:654] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:01.277674 2662 tablet_bootstrap.cc:654] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:01.278258 2662 tablet_bootstrap.cc:492] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6: No bootstrap required, opened a new log
I20260924 23:16:01.278254 2661 tablet_bootstrap.cc:492] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7: Bootstrap starting.
I20260924 23:16:01.278301 2662 ts_tablet_manager.cc:1397] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:01.278366 2663 tablet_bootstrap.cc:492] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001: No bootstrap required, opened a new log
I20260924 23:16:01.278407 2663 ts_tablet_manager.cc:1397] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:01.278462 2662 raft_consensus.cc:348] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.278527 2662 raft_consensus.cc:374] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:01.278546 2662 raft_consensus.cc:729] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d239f6eb754359870bc71bb97f81c6, State: Initialized, Role: FOLLOWER
I20260924 23:16:01.278546 2663 raft_consensus.cc:348] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.278586 2662 consensus_queue.cc:260] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.278616 2663 raft_consensus.cc:374] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:01.278611 2661 tablet_bootstrap.cc:654] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:01.278635 2663 raft_consensus.cc:729] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb0e3c157e84ffbb575f28ba8272001, State: Initialized, Role: FOLLOWER
I20260924 23:16:01.278671 2662 ts_tablet_manager.cc:1428] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:01.278674 2663 consensus_queue.cc:260] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.278751 2663 ts_tablet_manager.cc:1428] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:01.278904 2669 tablet_replica.cc:439] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } }
I20260924 23:16:01.278980 2669 tablet_replica.cc:442] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:01.278975 2667 tablet_replica.cc:439] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } }
I20260924 23:16:01.279055 2667 tablet_replica.cc:442] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:01.279335 2661 tablet_bootstrap.cc:492] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7: No bootstrap required, opened a new log
I20260924 23:16:01.279414 2661 ts_tablet_manager.cc:1397] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:01.279560 2661 raft_consensus.cc:348] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.279615 2661 raft_consensus.cc:374] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:01.279632 2661 raft_consensus.cc:729] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78cb8af4b9e344da8bee15a8566028e7, State: Initialized, Role: FOLLOWER
I20260924 23:16:01.279671 2661 consensus_queue.cc:260] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.279753 2661 ts_tablet_manager.cc:1428] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:01.279776 2671 tablet_replica.cc:439] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } }
I20260924 23:16:01.279871 2671 tablet_replica.cc:442] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 23:16:01.293783 2639 tablet.cc:2369] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:16:01.376308 2667 raft_consensus.cc:482] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:01.376444 2667 raft_consensus.cc:504] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.376750 2667 leader_election.cc:288] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33d239f6eb754359870bc71bb97f81c6 (127.1.4.193:36807), 78cb8af4b9e344da8bee15a8566028e7 (127.1.4.195:38981)
I20260924 23:16:01.379638 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.376993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:51961 (local address 127.1.4.193:36807)
0924 23:16:01.377047 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.377049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.377167 (+ 118us) server_negotiation.cc:407] Connection header received
0924 23:16:01.377216 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.377217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.377225 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.377289 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:01.377522 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.378001 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.378971 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.379310 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.379390 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.379410 (+ 20us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.379531 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.379570 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 23:16:01.379669 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.377000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:47727 (local address 127.1.4.195:38981)
0924 23:16:01.377090 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.377092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.377100 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:01.377130 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.377131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.377139 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.377195 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:01.377359 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.377920 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.378979 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.379355 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.379440 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.379449 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.379575 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.379612 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:16:01.379657 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.376973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:36807 (local address 127.1.4.194:51961)
0924 23:16:01.377105 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:01.377113 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:01.377160 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:01.377291 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:01.377292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:01.377293 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:01.377411 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.377413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.378081 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.378082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.378892 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.378894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.379399 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.379400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.379466 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.379471 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:16:01.379512 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 23:16:01.379554 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260924 23:16:01.379658 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.376917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:38981 (local address 127.1.4.194:47727)
0924 23:16:01.376974 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:01.376985 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:01.377028 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:01.377213 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:01.377214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:01.377216 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:01.377291 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.377294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.378004 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.378004 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.378886 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.378888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.379445 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.379446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.379504 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.379508 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:01.379561 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 23:16:01.379598 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":2}
I20260924 23:16:01.380014 2312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bab86a884ca408f9ce5d0022076ed27" candidate_uuid: "5cb0e3c157e84ffbb575f28ba8272001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d239f6eb754359870bc71bb97f81c6" is_pre_election: true
I20260924 23:16:01.379977 2592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bab86a884ca408f9ce5d0022076ed27" candidate_uuid: "5cb0e3c157e84ffbb575f28ba8272001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78cb8af4b9e344da8bee15a8566028e7" is_pre_election: true
I20260924 23:16:01.380107 2312 raft_consensus.cc:2454] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb0e3c157e84ffbb575f28ba8272001 in term 0.
I20260924 23:16:01.380123 2592 raft_consensus.cc:2454] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb0e3c157e84ffbb575f28ba8272001 in term 0.
I20260924 23:16:01.380328 2385 leader_election.cc:302] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb0e3c157e84ffbb575f28ba8272001, 78cb8af4b9e344da8bee15a8566028e7; no voters:
I20260924 23:16:01.380537 2667 raft_consensus.cc:2781] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:01.380570 2667 raft_consensus.cc:482] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:01.380590 2667 raft_consensus.cc:3037] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:01.381502 2667 raft_consensus.cc:504] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.381651 2667 leader_election.cc:288] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [CANDIDATE]: Term 1 election: Requested vote from peers 33d239f6eb754359870bc71bb97f81c6 (127.1.4.193:36807), 78cb8af4b9e344da8bee15a8566028e7 (127.1.4.195:38981)
I20260924 23:16:01.381805 2592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bab86a884ca408f9ce5d0022076ed27" candidate_uuid: "5cb0e3c157e84ffbb575f28ba8272001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78cb8af4b9e344da8bee15a8566028e7"
I20260924 23:16:01.381891 2592 raft_consensus.cc:3037] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:01.381866 2312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bab86a884ca408f9ce5d0022076ed27" candidate_uuid: "5cb0e3c157e84ffbb575f28ba8272001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d239f6eb754359870bc71bb97f81c6"
I20260924 23:16:01.382025 2312 raft_consensus.cc:3037] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:01.382704 2312 raft_consensus.cc:2454] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb0e3c157e84ffbb575f28ba8272001 in term 1.
I20260924 23:16:01.382704 2592 raft_consensus.cc:2454] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb0e3c157e84ffbb575f28ba8272001 in term 1.
I20260924 23:16:01.382972 2385 leader_election.cc:302] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb0e3c157e84ffbb575f28ba8272001, 78cb8af4b9e344da8bee15a8566028e7; no voters:
I20260924 23:16:01.383075 2667 raft_consensus.cc:2781] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:01.383250 2667 raft_consensus.cc:686] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [term 1 LEADER]: Becoming Leader. State: Replica: 5cb0e3c157e84ffbb575f28ba8272001, State: Running, Role: LEADER
I20260924 23:16:01.383354 2667 consensus_queue.cc:237] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } }
I20260924 23:16:01.383781 2701 tablet_replica.cc:439] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001: TxnStatusTablet state changed. Reason: New leader 5cb0e3c157e84ffbb575f28ba8272001. Latest consensus state: current_term: 1 leader_uuid: "5cb0e3c157e84ffbb575f28ba8272001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } }
I20260924 23:16:01.383872 2701 tablet_replica.cc:442] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:16:01.384155 2703 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 23:16:01.384191 2154 catalog_manager.cc:5495] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cb0e3c157e84ffbb575f28ba8272001 (127.1.4.194). New cstate: current_term: 1 leader_uuid: "5cb0e3c157e84ffbb575f28ba8272001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } health_report { overall_health: UNKNOWN } } }
I20260924 23:16:01.384285 2703 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 23:16:01.399626 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.397176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:42275 (local address 127.0.0.1:46360)
0924 23:16:01.397249 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:01.397261 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:01.397309 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:01.397470 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:01.397470 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:01.397472 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:16:01.397526 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.397529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.398163 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.398164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.398768 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.398770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.399101 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.399102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.399167 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.399183 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:16:01.399445 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:16:01.399454 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 23:16:01.399496 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 23:16:01.399546 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 23:16:01.399622 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.397250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.1.4.194:42275)
0924 23:16:01.397299 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.397301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.397310 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:01.397381 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.397382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.397402 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.397463 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:01.397578 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.398065 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.398844 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.399033 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.399089 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.399261 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:01.399386 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:01.399448 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.399499 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.399519 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 23:16:01.400856 2312 raft_consensus.cc:1261] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6 [term 1 FOLLOWER]: Refusing update from remote peer 5cb0e3c157e84ffbb575f28ba8272001: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:01.400969 2592 raft_consensus.cc:1261] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7 [term 1 FOLLOWER]: Refusing update from remote peer 5cb0e3c157e84ffbb575f28ba8272001: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:01.401147 2667 consensus_queue.cc:1035] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:01.401252 2701 consensus_queue.cc:1035] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:01.401834 2669 tablet_replica.cc:439] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6: TxnStatusTablet state changed. Reason: New leader 5cb0e3c157e84ffbb575f28ba8272001. Latest consensus state: current_term: 1 leader_uuid: "5cb0e3c157e84ffbb575f28ba8272001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } }
I20260924 23:16:01.401927 2669 tablet_replica.cc:442] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:01.401932 2671 tablet_replica.cc:439] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7: TxnStatusTablet state changed. Reason: New leader 5cb0e3c157e84ffbb575f28ba8272001. Latest consensus state: current_term: 1 leader_uuid: "5cb0e3c157e84ffbb575f28ba8272001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } }
I20260924 23:16:01.402015 2671 tablet_replica.cc:442] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:01.402663 2667 tablet_replica.cc:439] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cb0e3c157e84ffbb575f28ba8272001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } }
I20260924 23:16:01.402745 2667 tablet_replica.cc:442] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:16:01.402977 2669 tablet_replica.cc:439] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cb0e3c157e84ffbb575f28ba8272001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } }
I20260924 23:16:01.403057 2669 tablet_replica.cc:442] T 5bab86a884ca408f9ce5d0022076ed27 P 33d239f6eb754359870bc71bb97f81c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:01.408401 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.406184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:42275 (local address 127.1.4.195:38205)
0924 23:16:01.406262 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:01.406275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:01.406322 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:01.406422 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:01.406423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:01.406423 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:16:01.406472 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.406473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.407054 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.407055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.407692 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.407695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.408017 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.408018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.408071 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.408083 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:16:01.408243 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:16:01.408245 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:16:01.408286 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 23:16:01.408338 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 23:16:01.408385 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.406242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:38205 (local address 127.1.4.194:42275)
0924 23:16:01.406282 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.406284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.406308 (+ 24us) server_negotiation.cc:407] Connection header received
0924 23:16:01.406366 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.406367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.406377 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.406421 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:01.406525 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.406981 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.407762 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.407961 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.408013 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.408166 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:01.408197 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:01.408236 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.408294 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.408312 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:16:01.410432 2312 raft_consensus.cc:1261] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6 [term 1 FOLLOWER]: Refusing update from remote peer 78cb8af4b9e344da8bee15a8566028e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:01.410445 2452 raft_consensus.cc:1261] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001 [term 1 FOLLOWER]: Refusing update from remote peer 78cb8af4b9e344da8bee15a8566028e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:01.410652 2679 consensus_queue.cc:1035] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:01.410763 2671 consensus_queue.cc:1035] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:01.411144 2708 tablet_replica.cc:439] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cb0e3c157e84ffbb575f28ba8272001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } }
I20260924 23:16:01.411288 2708 tablet_replica.cc:442] T 5bab86a884ca408f9ce5d0022076ed27 P 5cb0e3c157e84ffbb575f28ba8272001: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:16:01.411661 2671 tablet_replica.cc:439] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cb0e3c157e84ffbb575f28ba8272001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33d239f6eb754359870bc71bb97f81c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36807 } } peers { permanent_uuid: "5cb0e3c157e84ffbb575f28ba8272001" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42275 } } peers { permanent_uuid: "78cb8af4b9e344da8bee15a8566028e7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38981 } } }
I20260924 23:16:01.411741 2671 tablet_replica.cc:442] T 5bab86a884ca408f9ce5d0022076ed27 P 78cb8af4b9e344da8bee15a8566028e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:01.412889 2676 mvcc.cc:205] Tried to move back new op lower bound from 7333035054735708160 to 7333035054032019456. Current Snapshot: MvccSnapshot[applied={T|T < 7333035054735708160}]
I20260924 23:16:01.425599 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.422789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:47709 (local address 127.1.4.195:38981)
0924 23:16:01.422831 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.422832 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.422983 (+ 151us) server_negotiation.cc:407] Connection header received
0924 23:16:01.423045 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.423046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.423070 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.423124 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:01.423337 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.423887 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.424975 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.425153 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.425199 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.425367 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:01.425391 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:01.425421 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.425473 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.425492 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:16:01.425623 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.422754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:38981 (local address 127.1.4.194:47709)
0924 23:16:01.422922 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:01.422932 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:01.422993 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:01.423175 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:01.423176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:01.423177 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:16:01.423252 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.423254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.423953 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.423954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.424901 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:01.424904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.425212 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.425213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:01.425279 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.425299 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:16:01.425424 (+ 125us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:16:01.425425 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 23:16:01.425472 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 23:16:01.425515 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":102,"threads_started":1}
I20260924 23:16:01.436123 2154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:01.436231 2154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:01.436796 2154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5d00a6e848c43db884e9b89271b47a0: Sending DeleteTablet for 3 replicas of tablet dfa4013f8f934ecf8475ccc0cb9923f4
I20260924 23:16:01.437012 2292 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa4013f8f934ecf8475ccc0cb9923f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:01 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:60076
I20260924 23:16:01.437055 2432 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa4013f8f934ecf8475ccc0cb9923f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:01 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:46354
I20260924 23:16:01.437085 2572 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa4013f8f934ecf8475ccc0cb9923f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:01 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:47996
I20260924 23:16:01.437140 2686 tablet_replica.cc:330] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6: stopping tablet replica
I20260924 23:16:01.437139 2685 tablet_replica.cc:330] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001: stopping tablet replica
I20260924 23:16:01.437183 2685 raft_consensus.cc:2229] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:01.437184 2686 raft_consensus.cc:2229] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:01.437218 2686 raft_consensus.cc:2258] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:01.437218 2685 raft_consensus.cc:2258] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:01.437282 2684 tablet_replica.cc:330] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7: stopping tablet replica
I20260924 23:16:01.437328 2684 raft_consensus.cc:2229] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:16:01.437387 2684 raft_consensus.cc:2258] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:01.437722 2686 ts_tablet_manager.cc:1905] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:01.437723 2685 ts_tablet_manager.cc:1905] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:01.437724 2684 ts_tablet_manager.cc:1905] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:01.439304 2684 ts_tablet_manager.cc:1918] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:16:01.439317 2685 ts_tablet_manager.cc:1918] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:16:01.439383 2684 log.cc:1196] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/dfa4013f8f934ecf8475ccc0cb9923f4
I20260924 23:16:01.439406 2685 log.cc:1196] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/dfa4013f8f934ecf8475ccc0cb9923f4
I20260924 23:16:01.439459 2686 ts_tablet_manager.cc:1918] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:16:01.439502 2686 log.cc:1196] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291755455739-1043-0/minicluster-data/ts-0/wal/wals/dfa4013f8f934ecf8475ccc0cb9923f4
I20260924 23:16:01.439617 2685 ts_tablet_manager.cc:1939] T dfa4013f8f934ecf8475ccc0cb9923f4 P 5cb0e3c157e84ffbb575f28ba8272001: Deleting consensus metadata
I20260924 23:16:01.439682 2686 ts_tablet_manager.cc:1939] T dfa4013f8f934ecf8475ccc0cb9923f4 P 33d239f6eb754359870bc71bb97f81c6: Deleting consensus metadata
I20260924 23:16:01.439689 2684 ts_tablet_manager.cc:1939] T dfa4013f8f934ecf8475ccc0cb9923f4 P 78cb8af4b9e344da8bee15a8566028e7: Deleting consensus metadata
I20260924 23:16:01.439985 2139 catalog_manager.cc:4841] TS 33d239f6eb754359870bc71bb97f81c6 (127.1.4.193:36807): tablet dfa4013f8f934ecf8475ccc0cb9923f4 (table test-table [id=a0bac1399a914500a962262333e58b5f]) successfully deleted
I20260924 23:16:01.440136 2140 catalog_manager.cc:4841] TS 5cb0e3c157e84ffbb575f28ba8272001 (127.1.4.194:42275): tablet dfa4013f8f934ecf8475ccc0cb9923f4 (table test-table [id=a0bac1399a914500a962262333e58b5f]) successfully deleted
I20260924 23:16:01.440578 2141 catalog_manager.cc:4841] TS 78cb8af4b9e344da8bee15a8566028e7 (127.1.4.195:38981): tablet dfa4013f8f934ecf8475ccc0cb9923f4 (table test-table [id=a0bac1399a914500a962262333e58b5f]) successfully deleted
Sep 24 23:16:01 dist-test-slave-s1hw krb5kdc[2110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291761, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.4.193@KRBTEST.COM
I20260924 23:16:01.448666 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.439207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.1.4.193:36807)
0924 23:16:01.439248 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.439250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.439976 (+ 726us) server_negotiation.cc:407] Connection header received
0924 23:16:01.440068 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.440069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.440079 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.440108 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:01.440765 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.441522 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.442301 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.442597 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.442668 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.445610 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:01.445621 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:01.445622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:01.445630 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:01.447600 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:01.447918 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:01.447923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:01.447923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:01.448002 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:01.448143 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:01.448144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:01.448144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:01.448321 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:01.448458 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.448494 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.448576 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 23:16:01.448953 2352 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60092
I20260924 23:16:01.456408 2204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:01.449886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.1.4.254:42509)
0924 23:16:01.449929 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:01.449931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:01.449940 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:01.450002 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:01.450003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:01.450012 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:01.450091 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:01.450711 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.451294 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.452326 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:01.452520 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:01.452577 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:01.454006 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:01.454007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:01.454007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:01.454014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:01.455336 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:01.455716 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:01.455717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:01.455717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:01.455771 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:01.455948 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:01.455949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:01.455949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:01.456084 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:01.456155 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:01.456277 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 23:16:01.456334 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 23:16:01.456773 2154 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53936
I20260924 23:16:01.457473 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 2227
I20260924 23:16:01.462201 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 2367
I20260924 23:16:01.467051 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 2507
I20260924 23:16:01.471985 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 2123
2026-09-24T23:16:01Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2243 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:16:01 dist-test-slave-s1hw krb5kdc[2723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:16:01 dist-test-slave-s1hw krb5kdc[2723](info): set up 2 sockets
Sep 24 23:16:01 dist-test-slave-s1hw krb5kdc[2723](info): commencing operation
krb5kdc: starting...
W20260924 23:16:02.427039 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:16:02 dist-test-slave-s1hw krb5kdc[2723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:16:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:16:02.542173 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38247
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:46805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:38247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:02.620399 2736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:02.620622 2736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:02.620652 2736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:02.621826 2736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:02.621872 2736 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 23:16:02.621898 2736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:02.621915 2736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:02.621930 2736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:02.621948 2736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:02.623477 2736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:46805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:38247
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:02.623733 2736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:02.623950 2736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:02.628641 2744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:02.628643 2742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:02.628633 2741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:02.628844 2736 server_base.cc:1034] running on GCE node
I20260924 23:16:02.629550 2736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:02.629846 2736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:02.631104 2736 hybrid_clock.cc:648] HybridClock initialized: now 1790291762631088 us; error 38 us; skew 500 ppm
Sep 24 23:16:02 dist-test-slave-s1hw krb5kdc[2723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291762, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:02.633559 2736 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:16:02.634006 2736 webserver.cc:460] Webserver started at http://127.1.4.254:43251/ using document root <none> and password file <none>
I20260924 23:16:02.634197 2736 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:02.634243 2736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:02.634366 2736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:02.635491 2736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "c2c224c3dedb4eab8516e3d1c342e3dc"
format_stamp: "Formatted at 2026-09-24 23:16:02 on dist-test-slave-s1hw"
server_key: "e3e18dccffb532fb0c5c9b653e6b5041"
server_key_iv: "cd9ecd946f2235b9118a40e0f931c599"
server_key_version: "encryptionkey@0"
I20260924 23:16:02.635834 2736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "c2c224c3dedb4eab8516e3d1c342e3dc"
format_stamp: "Formatted at 2026-09-24 23:16:02 on dist-test-slave-s1hw"
server_key: "e3e18dccffb532fb0c5c9b653e6b5041"
server_key_iv: "cd9ecd946f2235b9118a40e0f931c599"
server_key_version: "encryptionkey@0"
I20260924 23:16:02.637223 2736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:16:02.637858 2751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:02.638000 2736 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:02.638082 2736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "c2c224c3dedb4eab8516e3d1c342e3dc"
format_stamp: "Formatted at 2026-09-24 23:16:02 on dist-test-slave-s1hw"
server_key: "e3e18dccffb532fb0c5c9b653e6b5041"
server_key_iv: "cd9ecd946f2235b9118a40e0f931c599"
server_key_version: "encryptionkey@0"
I20260924 23:16:02.638137 2736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:02.656544 2736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:02.658836 2736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:02.659004 2736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:02.663866 2736 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:38247
I20260924 23:16:02.663915 2813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:38247 every 8 connection(s)
I20260924 23:16:02.664237 2736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:02.664820 2814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:02.666515 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 2736
I20260924 23:16:02.666622 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:02.666782 1043 external_mini_cluster.cc:1428] Setting key c9cba7e6d59f18d12676b14f14417a6b
I20260924 23:16:02.667387 2814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc: Bootstrap starting.
I20260924 23:16:02.667924 2814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:02.668182 2814 log.cc:824] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:02.668803 2814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc: No bootstrap required, opened a new log
I20260924 23:16:02.670614 2814 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c224c3dedb4eab8516e3d1c342e3dc" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38247 } }
I20260924 23:16:02.670775 2814 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:02.670806 2814 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c224c3dedb4eab8516e3d1c342e3dc, State: Initialized, Role: FOLLOWER
I20260924 23:16:02.670931 2814 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c224c3dedb4eab8516e3d1c342e3dc" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38247 } }
I20260924 23:16:02.671000 2814 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:02.671028 2814 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:02.671088 2814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:02.671756 2814 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c224c3dedb4eab8516e3d1c342e3dc" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38247 } }
I20260924 23:16:02.671861 2814 leader_election.cc:302] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2c224c3dedb4eab8516e3d1c342e3dc; no voters:
I20260924 23:16:02.672040 2814 leader_election.cc:288] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:02.672072 2819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:02.672214 2819 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [term 1 LEADER]: Becoming Leader. State: Replica: c2c224c3dedb4eab8516e3d1c342e3dc, State: Running, Role: LEADER
I20260924 23:16:02.672292 2814 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:02.672300 2819 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c224c3dedb4eab8516e3d1c342e3dc" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38247 } }
I20260924 23:16:02.672626 2819 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2c224c3dedb4eab8516e3d1c342e3dc. Latest consensus state: current_term: 1 leader_uuid: "c2c224c3dedb4eab8516e3d1c342e3dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c224c3dedb4eab8516e3d1c342e3dc" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38247 } } }
I20260924 23:16:02.672696 2819 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [sys.catalog]: This master's current role is: LEADER
Sep 24 23:16:02 dist-test-slave-s1hw krb5kdc[2723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
W20260924 23:16:02.674355 2835 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 23:16:02.673928 2821 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2c224c3dedb4eab8516e3d1c342e3dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c224c3dedb4eab8516e3d1c342e3dc" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38247 } } }
W20260924 23:16:02.674410 2835 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:16:02.674554 2823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:02.674719 2823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:02.674422 2821 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:02.676158 2823 catalog_manager.cc:1347] Generated new cluster ID: 73e171bd551f4137845edb0a358b4188
I20260924 23:16:02.676208 2823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:16:02.677316 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:02.667312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.1.4.254:38247)
0924 23:16:02.667519 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:02.667527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:02.667553 (+ 26us) server_negotiation.cc:407] Connection header received
0924 23:16:02.668004 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:02.668015 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:02.668072 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:02.668198 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:02.668884 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:02.669799 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:02.670520 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:02.670991 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:02.671086 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:02.673688 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:02.673692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:02.673693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:02.673706 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:02.675646 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:02.676034 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:02.676037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:02.676037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:02.676139 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:02.676343 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:02.676344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:02.676345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:02.676638 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:02.676776 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:02.676982 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 23:16:02.677113 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260924 23:16:02.717620 2823 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:02.718178 2823 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:02.724653 2823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc: Generated new TSK 0
I20260924 23:16:02.724835 2823 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:16:02.756544 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38247
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:46805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:02.837175 2840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:02.837375 2840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:02.837399 2840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:02.838490 2840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:02.838524 2840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:02.838543 2840 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 23:16:02.838553 2840 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:16:02.838569 2840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:16:02.839993 2840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:46805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.4.254:38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:02.840173 2840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:02.840346 2840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:02.845345 2845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:02.845345 2846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:02.845345 2848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:02.845633 2840 server_base.cc:1034] running on GCE node
I20260924 23:16:02.845793 2840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:02.845985 2840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:02.847136 2840 hybrid_clock.cc:648] HybridClock initialized: now 1790291762847122 us; error 26 us; skew 500 ppm
Sep 24 23:16:02 dist-test-slave-s1hw krb5kdc[2723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291762, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:02.849509 2840 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:16:02.849920 2840 webserver.cc:460] Webserver started at http://127.1.4.193:39201/ using document root <none> and password file <none>
I20260924 23:16:02.850133 2840 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:02.850178 2840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:02.850294 2840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:02.851399 2840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "6c0d2232d9f840369602e1f768053fd3"
format_stamp: "Formatted at 2026-09-24 23:16:02 on dist-test-slave-s1hw"
server_key: "6c24fad539a8bb98bbe8aaebfb0200fd"
server_key_iv: "64e462bbe09aaebb55e4d29902ecf36b"
server_key_version: "encryptionkey@0"
I20260924 23:16:02.851739 2840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "6c0d2232d9f840369602e1f768053fd3"
format_stamp: "Formatted at 2026-09-24 23:16:02 on dist-test-slave-s1hw"
server_key: "6c24fad539a8bb98bbe8aaebfb0200fd"
server_key_iv: "64e462bbe09aaebb55e4d29902ecf36b"
server_key_version: "encryptionkey@0"
I20260924 23:16:02.852892 2840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:02.853616 2855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:02.853772 2840 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:02.853845 2840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "6c0d2232d9f840369602e1f768053fd3"
format_stamp: "Formatted at 2026-09-24 23:16:02 on dist-test-slave-s1hw"
server_key: "6c24fad539a8bb98bbe8aaebfb0200fd"
server_key_iv: "64e462bbe09aaebb55e4d29902ecf36b"
server_key_version: "encryptionkey@0"
I20260924 23:16:02.853986 2840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:02.871774 2840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:02.873772 2840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:02.873890 2840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:02.874429 2840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:02.874470 2840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:02.874491 2840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:02.874503 2840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 23:16:02 dist-test-slave-s1hw krb5kdc[2723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291762, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:02.881034 2970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:44559 every 8 connection(s)
I20260924 23:16:02.880930 2840 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:44559
I20260924 23:16:02.881623 2840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:16:02.883849 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:02.874839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:45781 (local address 127.1.4.254:38247)
0924 23:16:02.874917 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:02.874920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:02.875087 (+ 167us) server_negotiation.cc:407] Connection header received
0924 23:16:02.875606 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:02.875608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:02.875621 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:02.875695 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:02.876336 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:02.877087 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:02.878554 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:02.878869 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:02.878935 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:02.881284 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:02.881290 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:02.881290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:02.881299 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:02.882657 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:02.883025 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:02.883025 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:02.883026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:02.883072 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:02.883249 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:02.883250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:02.883250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:02.883350 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:02.883432 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:02.883705 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 23:16:02.883783 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 23:16:02.883930 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:02.874806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38247 (local address 127.1.4.193:45781)
0924 23:16:02.874951 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:02.874991 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:02.875512 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:02.875775 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:02.875779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:02.876044 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:02.876246 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:02.876251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:02.877182 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:02.877183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:02.878461 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:02.878464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:02.878949 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:02.878950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:02.879103 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:02.879633 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:02.879641 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:02.881183 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:02.882734 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:02.882739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:02.882740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:02.882963 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:02.883120 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:02.883120 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:02.883120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:02.883208 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:02.883422 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:02.883427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:02.883553 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 23:16:02.883636 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 23:16:02.883754 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260924 23:16:02.884387 2971 heartbeater.cc:347] Connected to a master server at 127.1.4.254:38247
I20260924 23:16:02.884469 2971 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:02.884661 2971 heartbeater.cc:511] Master 127.1.4.254:38247 requested a full tablet report, sending...
I20260924 23:16:02.885258 2767 ts_manager.cc:194] Registered new tserver with Master: 6c0d2232d9f840369602e1f768053fd3 (127.1.4.193:44559)
I20260924 23:16:02.886116 2767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:45781
I20260924 23:16:02.890415 2977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:02.885345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38247 (local address 127.1.4.193:34289)
0924 23:16:02.885510 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:02.885549 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:02.885613 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:02.885721 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:02.885722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:02.885901 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:02.885958 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:02.885960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:02.886613 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:02.886614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:02.887235 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:02.887237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:02.887645 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:02.887646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:02.887701 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:02.888116 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:02.888119 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:02.888378 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:02.889741 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:02.889741 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:02.889742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:02.889875 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:02.890020 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:02.890020 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:02.890021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:02.890059 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:02.890210 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:02.890211 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:02.890240 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 23:16:02.890281 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 23:16:02.890345 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260924 23:16:02.890411 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:02.885389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:34289 (local address 127.1.4.254:38247)
0924 23:16:02.885451 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:02.885453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:02.885628 (+ 175us) server_negotiation.cc:407] Connection header received
0924 23:16:02.885654 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:02.885654 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:02.885664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:02.885716 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:02.886011 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:02.886538 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:02.887323 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:02.887569 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:02.887647 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:02.888459 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:02.888460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:02.888460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:02.888466 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:02.889684 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:02.889924 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:02.889925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:02.889925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:02.889981 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:02.890090 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:02.890091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:02.890091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:02.890176 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:02.890213 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:02.890297 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 23:16:02.890352 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 23:16:02.891196 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 2840
I20260924 23:16:02.891292 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:16:02.891494 1043 external_mini_cluster.cc:1428] Setting key 460ed0ff138291b291c280c1d1282ad7
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:16:02.915966 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38247
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:46805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:02.995458 2980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:02.995666 2980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:02.995698 2980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:02.996827 2980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:02.996871 2980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:02.996901 2980 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 23:16:02.996914 2980 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:16:02.996938 2980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:16:02.998375 2980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:46805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.4.254:38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:02.998615 2980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:02.998824 2980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:03.003647 2985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:03.003641 2986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:03.003728 2988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:03.003897 2980 server_base.cc:1034] running on GCE node
I20260924 23:16:03.004117 2980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:03.004328 2980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:03.005451 2980 hybrid_clock.cc:648] HybridClock initialized: now 1790291763005441 us; error 29 us; skew 500 ppm
Sep 24 23:16:03 dist-test-slave-s1hw krb5kdc[2723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291763, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:03.007970 2980 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:16:03.008442 2980 webserver.cc:460] Webserver started at http://127.1.4.194:45327/ using document root <none> and password file <none>
I20260924 23:16:03.008641 2980 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:03.008685 2980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:03.008791 2980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:03.009840 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "306cc64cbde348e0b72e3c588e8561cb"
format_stamp: "Formatted at 2026-09-24 23:16:03 on dist-test-slave-s1hw"
server_key: "0c9ed5197571f4ff454374276273e15d"
server_key_iv: "2de9c233f17e64d8db4ee7287f7f4def"
server_key_version: "encryptionkey@0"
I20260924 23:16:03.010179 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "306cc64cbde348e0b72e3c588e8561cb"
format_stamp: "Formatted at 2026-09-24 23:16:03 on dist-test-slave-s1hw"
server_key: "0c9ed5197571f4ff454374276273e15d"
server_key_iv: "2de9c233f17e64d8db4ee7287f7f4def"
server_key_version: "encryptionkey@0"
I20260924 23:16:03.011502 2980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 23:16:03.012244 2995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:03.012395 2980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:03.012477 2980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "306cc64cbde348e0b72e3c588e8561cb"
format_stamp: "Formatted at 2026-09-24 23:16:03 on dist-test-slave-s1hw"
server_key: "0c9ed5197571f4ff454374276273e15d"
server_key_iv: "2de9c233f17e64d8db4ee7287f7f4def"
server_key_version: "encryptionkey@0"
I20260924 23:16:03.012535 2980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:03.044477 2980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:03.046505 2980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:03.046643 2980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:03.047231 2980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:03.047273 2980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:03.047302 2980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:03.047333 2980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 23:16:03 dist-test-slave-s1hw krb5kdc[2723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291763, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:03.053962 2980 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:38977
I20260924 23:16:03.054039 3110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:38977 every 8 connection(s)
I20260924 23:16:03.054355 2980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:16:03.056274 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.047646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:40347 (local address 127.1.4.254:38247)
0924 23:16:03.047710 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.047712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.047897 (+ 185us) server_negotiation.cc:407] Connection header received
0924 23:16:03.048490 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.048491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.048504 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.048594 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:03.049185 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.049927 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.051023 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.051215 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.051270 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.053625 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:03.053626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:03.053627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:03.053634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:03.055124 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:03.055521 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:03.055522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:03.055522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:03.055604 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:03.055778 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:03.055778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:03.055778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:03.055871 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:03.055912 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.056142 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.056211 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 23:16:03.056458 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.047636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38247 (local address 127.1.4.194:40347)
0924 23:16:03.047781 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:03.047822 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:03.048381 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:03.048669 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:03.048674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:03.048921 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:03.049118 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.049123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.050032 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.050033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.050942 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.050944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.051299 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.051301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.051466 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.051950 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:03.051959 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:03.053524 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:03.055185 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:03.055190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:03.055192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:03.055464 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:03.055652 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:03.055653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:03.055653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:03.055742 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:03.055912 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:03.055916 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:03.056043 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 23:16:03.056125 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 23:16:03.056242 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260924 23:16:03.056887 3111 heartbeater.cc:347] Connected to a master server at 127.1.4.254:38247
I20260924 23:16:03.056965 3111 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:03.057138 3111 heartbeater.cc:511] Master 127.1.4.254:38247 requested a full tablet report, sending...
I20260924 23:16:03.057514 2768 ts_manager.cc:194] Registered new tserver with Master: 306cc64cbde348e0b72e3c588e8561cb (127.1.4.194:38977)
I20260924 23:16:03.057992 2768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:40347
I20260924 23:16:03.060964 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 2980
I20260924 23:16:03.061062 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:16:03.061241 1043 external_mini_cluster.cc:1428] Setting key 26b4ff335f5bded56f695e0d4859cb77
I20260924 23:16:03.062875 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.057890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:43929 (local address 127.1.4.254:38247)
0924 23:16:03.057925 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.057926 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.057994 (+ 68us) server_negotiation.cc:407] Connection header received
0924 23:16:03.058040 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.058040 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.058049 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.058104 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:03.058410 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.058857 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.059580 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.059757 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.059800 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.060601 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:03.060602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:03.060602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:03.060606 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:03.061882 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:03.062170 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:03.062171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:03.062172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:03.062223 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:03.062374 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:03.062375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:03.062375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:03.062503 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:03.062556 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.062715 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.062786 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:16:03.062902 3117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.057836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38247 (local address 127.1.4.194:43929)
0924 23:16:03.057945 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:03.057955 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:03.058004 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:03.058126 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:03.058127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:03.058278 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:03.058343 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.058345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.058937 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.058938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.059481 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.059482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.059804 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.059804 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.059853 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.060275 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:03.060280 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:03.060542 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:03.061946 (+ 1404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:03.061947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:03.061948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:03.062111 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:03.062272 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:03.062273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:03.062274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:03.062326 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:03.062584 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:03.062585 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:03.062626 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 23:16:03.062680 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 23:16:03.062790 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:16:03.084316 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38247
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:46805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:03.164119 3120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:03.164312 3120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:03.164343 3120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:03.165436 3120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:03.165473 3120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:03.165499 3120 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 23:16:03.165513 3120 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:16:03.165537 3120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:16:03.166929 3120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:46805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.4.254:38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:03.167147 3120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:03.167356 3120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:03.172219 3128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:03.172204 3126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:03.172230 3120 server_base.cc:1034] running on GCE node
W20260924 23:16:03.172189 3125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:03.172533 3120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:03.172793 3120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:03.173938 3120 hybrid_clock.cc:648] HybridClock initialized: now 1790291763173915 us; error 33 us; skew 500 ppm
Sep 24 23:16:03 dist-test-slave-s1hw krb5kdc[2723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291763, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:03.176342 3120 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:16:03.176816 3120 webserver.cc:460] Webserver started at http://127.1.4.195:36521/ using document root <none> and password file <none>
I20260924 23:16:03.177052 3120 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:03.177105 3120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:03.177237 3120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:03.178299 3120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "9b9dec66e98846e78cd730fe92c8a6b7"
format_stamp: "Formatted at 2026-09-24 23:16:03 on dist-test-slave-s1hw"
server_key: "d66ae51b7d50bf62e03c7a4b45753262"
server_key_iv: "9b68abe3123feda415f2a0a9ba2c057a"
server_key_version: "encryptionkey@0"
I20260924 23:16:03.178653 3120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "9b9dec66e98846e78cd730fe92c8a6b7"
format_stamp: "Formatted at 2026-09-24 23:16:03 on dist-test-slave-s1hw"
server_key: "d66ae51b7d50bf62e03c7a4b45753262"
server_key_iv: "9b68abe3123feda415f2a0a9ba2c057a"
server_key_version: "encryptionkey@0"
I20260924 23:16:03.180110 3120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:03.180850 3135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:03.181051 3120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:03.181139 3120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "9b9dec66e98846e78cd730fe92c8a6b7"
format_stamp: "Formatted at 2026-09-24 23:16:03 on dist-test-slave-s1hw"
server_key: "d66ae51b7d50bf62e03c7a4b45753262"
server_key_iv: "9b68abe3123feda415f2a0a9ba2c057a"
server_key_version: "encryptionkey@0"
I20260924 23:16:03.181198 3120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:03.193773 3120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:03.195971 3120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:03.196110 3120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:03.196661 3120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:03.196702 3120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:03.196734 3120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:03.196751 3120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 23:16:03 dist-test-slave-s1hw krb5kdc[2723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291763, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:03.203271 3120 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:34167
I20260924 23:16:03.203467 3250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:34167 every 8 connection(s)
I20260924 23:16:03.203702 3120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:16:03.205582 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.197143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:38315 (local address 127.1.4.254:38247)
0924 23:16:03.197190 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.197192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.197399 (+ 207us) server_negotiation.cc:407] Connection header received
0924 23:16:03.197905 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.197906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.197918 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.197970 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:03.198610 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.199352 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.200497 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.200710 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.200768 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.203042 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:03.203043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:03.203044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:03.203052 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:03.204469 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:03.204814 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:03.204815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:03.204816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:03.204868 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:03.205023 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:03.205023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:03.205023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:03.205120 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:03.205163 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.205445 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.205517 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:16:03.205664 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.197116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38247 (local address 127.1.4.195:38315)
0924 23:16:03.197285 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:03.197320 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:03.197809 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:03.198042 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:03.198056 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:03.198308 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:03.198493 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.198498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.199457 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.199458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.200426 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.200429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.200792 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.200793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.200931 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.201431 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:03.201441 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:03.202949 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:03.204538 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:03.204541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:03.204543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:03.204755 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:03.204905 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:03.204905 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:03.204905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:03.204988 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:03.205164 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:03.205168 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:03.205287 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 23:16:03.205374 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 23:16:03.205496 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260924 23:16:03.206178 3251 heartbeater.cc:347] Connected to a master server at 127.1.4.254:38247
I20260924 23:16:03.206251 3251 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:03.206427 3251 heartbeater.cc:511] Master 127.1.4.254:38247 requested a full tablet report, sending...
I20260924 23:16:03.206769 2767 ts_manager.cc:194] Registered new tserver with Master: 9b9dec66e98846e78cd730fe92c8a6b7 (127.1.4.195:34167)
I20260924 23:16:03.207211 2767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:38315
I20260924 23:16:03.208894 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 3120
I20260924 23:16:03.208987 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:16:03.209170 1043 external_mini_cluster.cc:1428] Setting key fc40cf31577a9548ca1650616f5f1848
I20260924 23:16:03.209928 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:16:03.212708 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.207178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:59315 (local address 127.1.4.254:38247)
0924 23:16:03.207218 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.207220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.207296 (+ 76us) server_negotiation.cc:407] Connection header received
0924 23:16:03.207347 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.207347 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.207357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.207425 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:03.207776 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.208260 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.209230 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.209494 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.209556 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.210510 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:03.210511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:03.210512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:03.210518 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:03.211746 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:03.212100 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:03.212101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:03.212101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:03.212147 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:03.212302 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:03.212303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:03.212303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:03.212390 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:03.212434 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.212570 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.212624 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:16:03.212734 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.207110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38247 (local address 127.1.4.195:59315)
0924 23:16:03.207231 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:03.207241 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:03.207294 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:03.207428 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:03.207429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:03.207616 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:03.207690 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.207693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.208366 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.208367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.209131 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.209134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.209553 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.209555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.209633 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.210060 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:03.210066 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:03.210434 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:03.211802 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:03.211803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:03.211803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:03.212013 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:03.212198 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:03.212198 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:03.212198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:03.212256 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:03.212437 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:03.212438 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:03.212483 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 23:16:03.212550 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 23:16:03.212647 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
Sep 24 23:16:03 dist-test-slave-s1hw krb5kdc[2723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 23:16:03 dist-test-slave-s1hw krb5kdc[2723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:03.231297 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.225224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.1.4.254:38247)
0924 23:16:03.225270 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.225271 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.225279 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:03.225301 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.225302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.225310 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.225361 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:03.225856 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.226372 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.227093 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.227266 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.227311 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.229340 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:03.229341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:03.229341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:03.229348 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:03.230445 (+ 1097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:03.230753 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:03.230754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:03.230754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:03.230795 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:03.230923 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:03.230923 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:03.230923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:03.231007 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:03.231107 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.231180 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.231231 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 23:16:03.232797 2767 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:16:03.234315 2767 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:03.240383 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.236425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:34167 (local address 127.0.0.1:46652)
0924 23:16:03.236542 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:03.236582 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:03.236645 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:03.236961 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:03.236965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:03.236967 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:03.237027 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.237029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.238036 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.238037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.239209 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.239211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.240088 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.240089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.240168 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.240183 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 23:16:03.240252 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 23:16:03.240281 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260924 23:16:03.240403 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.236672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46652 (local address 127.1.4.195:34167)
0924 23:16:03.236823 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.236828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.236837 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:03.236866 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.236867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.236879 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.236946 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:03.237095 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.237961 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.239299 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.240008 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.240077 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.240136 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.240272 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.240309 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260924 23:16:03.240386 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.236426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:38977 (local address 127.0.0.1:54150)
0924 23:16:03.236602 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:03.236612 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:03.236659 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:03.237386 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:03.237387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:03.237389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:03.237425 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.237427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.238608 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.238609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.239491 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.239493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.240112 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.240113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.240168 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.240184 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 23:16:03.240228 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:16:03.240279 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260924 23:16:03.240665 3273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.237013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.1.4.194:38977)
0924 23:16:03.237229 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.237233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.237241 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:03.237276 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.237278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.237291 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.237490 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:03.237544 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.238544 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.239556 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.240020 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.240423 (+ 403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.240496 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.240520 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.240560 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":113,"threads_started":1}
I20260924 23:16:03.240830 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.236427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:44559 (local address 127.0.0.1:35452)
0924 23:16:03.236654 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:03.236662 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:03.236699 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:03.237135 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:03.237136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:03.237138 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:03.237175 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.237176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.238449 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.238450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.239530 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.239532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.240433 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.240434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.240679 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.240685 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 23:16:03.240733 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 23:16:03.240769 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":29,"thread_start_us":91,"threads_started":1}
I20260924 23:16:03.241124 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.236850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.1.4.193:44559)
0924 23:16:03.236979 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.236984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.236995 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:16:03.237032 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.237034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.237049 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.237139 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:03.237274 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.238356 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.239613 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.240277 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.240336 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.240399 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.241014 (+ 615us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.241051 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20260924 23:16:03.241261 3185 tablet_service.cc:1467] Processing CreateTablet for tablet df96de9e26d44728a4a68c3048065532 (DEFAULT_TABLE table=test-table [id=ab7a73783f504095bb6a6e179a21f850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:03.241669 3185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df96de9e26d44728a4a68c3048065532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:03.241783 2905 tablet_service.cc:1467] Processing CreateTablet for tablet df96de9e26d44728a4a68c3048065532 (DEFAULT_TABLE table=test-table [id=ab7a73783f504095bb6a6e179a21f850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:03.242141 2905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df96de9e26d44728a4a68c3048065532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:03.243793 3040 tablet_service.cc:1467] Processing CreateTablet for tablet df96de9e26d44728a4a68c3048065532 (DEFAULT_TABLE table=test-table [id=ab7a73783f504095bb6a6e179a21f850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:03.244102 3040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df96de9e26d44728a4a68c3048065532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:03.244882 3275 tablet_bootstrap.cc:492] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3: Bootstrap starting.
I20260924 23:16:03.245503 3275 tablet_bootstrap.cc:654] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:03.245827 3275 log.cc:824] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:03.246567 3275 tablet_bootstrap.cc:492] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3: No bootstrap required, opened a new log
I20260924 23:16:03.246623 3275 ts_tablet_manager.cc:1397] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:03.247661 3274 tablet_bootstrap.cc:492] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7: Bootstrap starting.
I20260924 23:16:03.247749 3277 tablet_bootstrap.cc:492] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb: Bootstrap starting.
I20260924 23:16:03.248342 3277 tablet_bootstrap.cc:654] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:03.248420 3274 tablet_bootstrap.cc:654] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:03.248642 3277 log.cc:824] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:03.248692 3274 log.cc:824] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:03.248862 3275 raft_consensus.cc:348] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } }
I20260924 23:16:03.249032 3275 raft_consensus.cc:374] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:03.249070 3275 raft_consensus.cc:729] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0d2232d9f840369602e1f768053fd3, State: Initialized, Role: FOLLOWER
I20260924 23:16:03.249208 3275 consensus_queue.cc:260] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } }
I20260924 23:16:03.249384 3274 tablet_bootstrap.cc:492] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7: No bootstrap required, opened a new log
I20260924 23:16:03.249420 3277 tablet_bootstrap.cc:492] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb: No bootstrap required, opened a new log
I20260924 23:16:03.249451 3274 ts_tablet_manager.cc:1397] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:03.249475 3275 ts_tablet_manager.cc:1428] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:16:03.249480 3277 ts_tablet_manager.cc:1397] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:03.249594 2971 heartbeater.cc:503] Master 127.1.4.254:38247 was elected leader, sending a full tablet report...
I20260924 23:16:03.251230 3274 raft_consensus.cc:348] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } }
I20260924 23:16:03.251245 3277 raft_consensus.cc:348] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } }
I20260924 23:16:03.251354 3277 raft_consensus.cc:374] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:03.251412 3277 raft_consensus.cc:729] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306cc64cbde348e0b72e3c588e8561cb, State: Initialized, Role: FOLLOWER
I20260924 23:16:03.251353 3274 raft_consensus.cc:374] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:03.251442 3274 raft_consensus.cc:729] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9dec66e98846e78cd730fe92c8a6b7, State: Initialized, Role: FOLLOWER
I20260924 23:16:03.251518 3277 consensus_queue.cc:260] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } }
I20260924 23:16:03.251582 3274 consensus_queue.cc:260] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } }
I20260924 23:16:03.251767 3277 ts_tablet_manager.cc:1428] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 23:16:03.251796 3111 heartbeater.cc:503] Master 127.1.4.254:38247 was elected leader, sending a full tablet report...
I20260924 23:16:03.251845 3274 ts_tablet_manager.cc:1428] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 23:16:03.251922 3251 heartbeater.cc:503] Master 127.1.4.254:38247 was elected leader, sending a full tablet report...
W20260924 23:16:03.305460 3112 tablet.cc:2369] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:16:03.349248 3282 raft_consensus.cc:482] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:03.349383 3282 raft_consensus.cc:504] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } }
I20260924 23:16:03.349764 3282 leader_election.cc:288] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c0d2232d9f840369602e1f768053fd3 (127.1.4.193:44559), 306cc64cbde348e0b72e3c588e8561cb (127.1.4.194:38977)
I20260924 23:16:03.352730 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.350060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:49537 (local address 127.1.4.193:44559)
0924 23:16:03.350096 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.350098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.350132 (+ 34us) server_negotiation.cc:407] Connection header received
0924 23:16:03.350160 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.350161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.350173 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.350225 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:03.350376 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.350907 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.352090 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.352425 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.352483 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.352493 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.352631 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.352665 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:16:03.352730 3273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.350042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:49041 (local address 127.1.4.194:38977)
0924 23:16:03.350092 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.350094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.350229 (+ 135us) server_negotiation.cc:407] Connection header received
0924 23:16:03.350261 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.350262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.350273 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.350327 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:03.350509 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.351045 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.352078 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.352420 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.352477 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.352489 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.352628 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.352657 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 23:16:03.352746 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.349960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:44559 (local address 127.1.4.195:49537)
0924 23:16:03.350019 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:03.350041 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:03.350092 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:03.350239 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:03.350241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:03.350243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:03.350311 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.350313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.351079 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.351080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.352002 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.352004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.352490 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.352491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.352568 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.352571 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:03.352615 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:16:03.352657 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":36}
I20260924 23:16:03.352753 3283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.349978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:38977 (local address 127.1.4.195:49041)
0924 23:16:03.350159 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:03.350168 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:03.350221 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:03.350333 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:03.350334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:03.350335 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:03.350411 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.350413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.351124 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.351125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.352000 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.352002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.352496 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.352497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.352567 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.352570 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:03.352616 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 23:16:03.352655 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":1,"thread_start_us":116,"threads_started":1}
I20260924 23:16:03.353075 2925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df96de9e26d44728a4a68c3048065532" candidate_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0d2232d9f840369602e1f768053fd3" is_pre_election: true
I20260924 23:16:03.353122 3065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df96de9e26d44728a4a68c3048065532" candidate_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306cc64cbde348e0b72e3c588e8561cb" is_pre_election: true
I20260924 23:16:03.353228 2925 raft_consensus.cc:2454] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b9dec66e98846e78cd730fe92c8a6b7 in term 0.
I20260924 23:16:03.353256 3065 raft_consensus.cc:2454] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b9dec66e98846e78cd730fe92c8a6b7 in term 0.
I20260924 23:16:03.353430 3137 leader_election.cc:302] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c0d2232d9f840369602e1f768053fd3, 9b9dec66e98846e78cd730fe92c8a6b7; no voters:
I20260924 23:16:03.353595 3282 raft_consensus.cc:2781] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:03.353643 3282 raft_consensus.cc:482] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:03.353658 3282 raft_consensus.cc:3037] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:03.354372 3282 raft_consensus.cc:504] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } }
I20260924 23:16:03.354496 3282 leader_election.cc:288] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [CANDIDATE]: Term 1 election: Requested vote from peers 6c0d2232d9f840369602e1f768053fd3 (127.1.4.193:44559), 306cc64cbde348e0b72e3c588e8561cb (127.1.4.194:38977)
I20260924 23:16:03.354624 3065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df96de9e26d44728a4a68c3048065532" candidate_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306cc64cbde348e0b72e3c588e8561cb"
I20260924 23:16:03.354693 2925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df96de9e26d44728a4a68c3048065532" candidate_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0d2232d9f840369602e1f768053fd3"
I20260924 23:16:03.354709 3065 raft_consensus.cc:3037] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:03.354765 2925 raft_consensus.cc:3037] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:03.355494 2925 raft_consensus.cc:2454] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b9dec66e98846e78cd730fe92c8a6b7 in term 1.
I20260924 23:16:03.355496 3065 raft_consensus.cc:2454] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b9dec66e98846e78cd730fe92c8a6b7 in term 1.
I20260924 23:16:03.355652 3138 leader_election.cc:302] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306cc64cbde348e0b72e3c588e8561cb, 9b9dec66e98846e78cd730fe92c8a6b7; no voters:
I20260924 23:16:03.355755 3282 raft_consensus.cc:2781] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:03.355914 3282 raft_consensus.cc:686] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 LEADER]: Becoming Leader. State: Replica: 9b9dec66e98846e78cd730fe92c8a6b7, State: Running, Role: LEADER
I20260924 23:16:03.356011 3282 consensus_queue.cc:237] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } }
I20260924 23:16:03.356757 2768 catalog_manager.cc:5495] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b9dec66e98846e78cd730fe92c8a6b7 (127.1.4.195). New cstate: current_term: 1 leader_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } health_report { overall_health: UNKNOWN } } }
W20260924 23:16:03.382349 2972 tablet.cc:2369] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 23:16:03 dist-test-slave-s1hw krb5kdc[2723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291762, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:03.422852 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.416146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38247 (local address 127.0.0.1:58860)
0924 23:16:03.416297 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:03.416308 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:03.416370 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:03.416509 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:03.416510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:03.416809 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:03.416855 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.416857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.417479 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.417480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.418044 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.418046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.418438 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.418439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.418498 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.419107 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:03.419111 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:03.420495 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:03.421846 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:03.421854 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:03.421855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:03.422076 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:03.422229 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:03.422230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:03.422230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:03.422291 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:03.422521 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:03.422528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:03.422573 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 23:16:03.422637 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 23:16:03.422742 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260924 23:16:03.422863 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.416163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58860 (local address 127.1.4.254:38247)
0924 23:16:03.416210 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.416213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.416363 (+ 150us) server_negotiation.cc:407] Connection header received
0924 23:16:03.416430 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.416430 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.416439 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.416497 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:03.416919 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.417393 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.418148 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.418339 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.418413 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.420574 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:03.420576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:03.420576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:03.420582 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:03.421769 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:03.422136 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:03.422136 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:03.422137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:03.422182 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:03.422340 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:03.422340 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:03.422340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:03.422430 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:03.422474 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.422663 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.422782 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":26}
I20260924 23:16:03.425215 2768 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:58860:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 23:16:03.425459 2768 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:03.427026 3040 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1481371c21426a9e054afa3624f1b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b9cacf9e37a94fa5b0807eebff0db2a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:16:03.426997 2905 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1481371c21426a9e054afa3624f1b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b9cacf9e37a94fa5b0807eebff0db2a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:16:03.427080 3185 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1481371c21426a9e054afa3624f1b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b9cacf9e37a94fa5b0807eebff0db2a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:16:03.427176 2905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1481371c21426a9e054afa3624f1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:03.427176 3040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1481371c21426a9e054afa3624f1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:03.427256 3185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1481371c21426a9e054afa3624f1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:03.428442 3274 tablet_bootstrap.cc:492] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7: Bootstrap starting.
I20260924 23:16:03.428608 3275 tablet_bootstrap.cc:492] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3: Bootstrap starting.
I20260924 23:16:03.428872 3274 tablet_bootstrap.cc:654] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:03.428895 3275 tablet_bootstrap.cc:654] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:03.429147 3277 tablet_bootstrap.cc:492] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb: Bootstrap starting.
I20260924 23:16:03.429432 3275 tablet_bootstrap.cc:492] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3: No bootstrap required, opened a new log
I20260924 23:16:03.429482 3275 ts_tablet_manager.cc:1397] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:03.429625 3275 raft_consensus.cc:348] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.429692 3275 raft_consensus.cc:374] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:03.429706 3275 raft_consensus.cc:729] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0d2232d9f840369602e1f768053fd3, State: Initialized, Role: FOLLOWER
I20260924 23:16:03.429754 3277 tablet_bootstrap.cc:654] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:03.429754 3275 consensus_queue.cc:260] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.429859 3275 ts_tablet_manager.cc:1428] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:03.429999 3280 tablet_replica.cc:439] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } }
I20260924 23:16:03.430083 3280 tablet_replica.cc:442] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:03.430540 3277 tablet_bootstrap.cc:492] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb: No bootstrap required, opened a new log
I20260924 23:16:03.430524 3274 tablet_bootstrap.cc:492] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7: No bootstrap required, opened a new log
I20260924 23:16:03.430588 3277 ts_tablet_manager.cc:1397] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 23:16:03.430588 3274 ts_tablet_manager.cc:1397] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 23:16:03.430747 3277 raft_consensus.cc:348] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.430747 3274 raft_consensus.cc:348] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.430819 3274 raft_consensus.cc:374] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:03.430819 3277 raft_consensus.cc:374] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:03.430840 3277 raft_consensus.cc:729] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306cc64cbde348e0b72e3c588e8561cb, State: Initialized, Role: FOLLOWER
I20260924 23:16:03.430840 3274 raft_consensus.cc:729] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9dec66e98846e78cd730fe92c8a6b7, State: Initialized, Role: FOLLOWER
I20260924 23:16:03.430883 3277 consensus_queue.cc:260] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.430881 3274 consensus_queue.cc:260] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.430962 3277 ts_tablet_manager.cc:1428] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:03.430965 3274 ts_tablet_manager.cc:1428] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 23:16:03.431092 3281 tablet_replica.cc:439] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } }
I20260924 23:16:03.431169 3281 tablet_replica.cc:442] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:03.431277 3282 tablet_replica.cc:439] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } }
I20260924 23:16:03.431345 3282 tablet_replica.cc:442] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 23:16:03.454242 3252 tablet.cc:2369] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:16:03.462270 3282 raft_consensus.cc:482] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:03.462361 3282 raft_consensus.cc:504] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.462512 3282 leader_election.cc:288] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c0d2232d9f840369602e1f768053fd3 (127.1.4.193:44559), 306cc64cbde348e0b72e3c588e8561cb (127.1.4.194:38977)
I20260924 23:16:03.462733 3065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1481371c21426a9e054afa3624f1b8" candidate_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306cc64cbde348e0b72e3c588e8561cb" is_pre_election: true
I20260924 23:16:03.462738 2925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1481371c21426a9e054afa3624f1b8" candidate_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0d2232d9f840369602e1f768053fd3" is_pre_election: true
I20260924 23:16:03.462818 3065 raft_consensus.cc:2454] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b9dec66e98846e78cd730fe92c8a6b7 in term 0.
I20260924 23:16:03.462833 2925 raft_consensus.cc:2454] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b9dec66e98846e78cd730fe92c8a6b7 in term 0.
I20260924 23:16:03.462962 3138 leader_election.cc:302] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306cc64cbde348e0b72e3c588e8561cb, 9b9dec66e98846e78cd730fe92c8a6b7; no voters:
I20260924 23:16:03.463110 3282 raft_consensus.cc:2781] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:03.463153 3282 raft_consensus.cc:482] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:03.463172 3282 raft_consensus.cc:3037] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:03.463846 3282 raft_consensus.cc:504] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.463977 3282 leader_election.cc:288] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [CANDIDATE]: Term 1 election: Requested vote from peers 6c0d2232d9f840369602e1f768053fd3 (127.1.4.193:44559), 306cc64cbde348e0b72e3c588e8561cb (127.1.4.194:38977)
I20260924 23:16:03.464121 2925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1481371c21426a9e054afa3624f1b8" candidate_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0d2232d9f840369602e1f768053fd3"
I20260924 23:16:03.464197 2925 raft_consensus.cc:3037] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:03.464169 3065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1481371c21426a9e054afa3624f1b8" candidate_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306cc64cbde348e0b72e3c588e8561cb"
I20260924 23:16:03.464241 3065 raft_consensus.cc:3037] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:03.464782 2925 raft_consensus.cc:2454] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b9dec66e98846e78cd730fe92c8a6b7 in term 1.
I20260924 23:16:03.464887 3065 raft_consensus.cc:2454] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b9dec66e98846e78cd730fe92c8a6b7 in term 1.
I20260924 23:16:03.464951 3137 leader_election.cc:302] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c0d2232d9f840369602e1f768053fd3, 9b9dec66e98846e78cd730fe92c8a6b7; no voters:
I20260924 23:16:03.465056 3282 raft_consensus.cc:2781] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:03.465106 3282 raft_consensus.cc:686] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 LEADER]: Becoming Leader. State: Replica: 9b9dec66e98846e78cd730fe92c8a6b7, State: Running, Role: LEADER
I20260924 23:16:03.465153 3282 consensus_queue.cc:237] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.465461 3284 tablet_replica.cc:439] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7: TxnStatusTablet state changed. Reason: New leader 9b9dec66e98846e78cd730fe92c8a6b7. Latest consensus state: current_term: 1 leader_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } }
I20260924 23:16:03.465536 3284 tablet_replica.cc:442] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:16:03.465758 3296 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 23:16:03.465749 2768 catalog_manager.cc:5495] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b9dec66e98846e78cd730fe92c8a6b7 (127.1.4.195). New cstate: current_term: 1 leader_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } health_report { overall_health: HEALTHY } } }
I20260924 23:16:03.465875 3296 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 23:16:03.479727 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.477253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:34167 (local address 127.0.0.1:46658)
0924 23:16:03.477321 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:03.477334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:03.477396 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:03.477606 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:03.477607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:03.477609 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:16:03.477667 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.477669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.478260 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.478261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.478818 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.478820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.479176 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.479177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.479236 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.479250 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:16:03.479550 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:16:03.479559 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 23:16:03.479609 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 23:16:03.479659 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 23:16:03.479717 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.477338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46658 (local address 127.1.4.195:34167)
0924 23:16:03.477424 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.477425 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.477459 (+ 34us) server_negotiation.cc:407] Connection header received
0924 23:16:03.477518 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.477519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.477539 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.477589 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:03.477723 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.478191 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.478892 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.479110 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.479164 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.479335 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:03.479493 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:03.479566 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.479615 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.479635 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 23:16:03.481704 3065 raft_consensus.cc:1261] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Refusing update from remote peer 9b9dec66e98846e78cd730fe92c8a6b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:03.481709 2925 raft_consensus.cc:1261] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3 [term 1 FOLLOWER]: Refusing update from remote peer 9b9dec66e98846e78cd730fe92c8a6b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:03.482023 3284 consensus_queue.cc:1035] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:03.482165 3282 consensus_queue.cc:1035] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:03.483466 3281 tablet_replica.cc:439] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb: TxnStatusTablet state changed. Reason: New leader 9b9dec66e98846e78cd730fe92c8a6b7. Latest consensus state: current_term: 1 leader_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } }
I20260924 23:16:03.483596 3281 tablet_replica.cc:442] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:03.483712 3280 tablet_replica.cc:439] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3: TxnStatusTablet state changed. Reason: New leader 9b9dec66e98846e78cd730fe92c8a6b7. Latest consensus state: current_term: 1 leader_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } }
I20260924 23:16:03.483788 3280 tablet_replica.cc:442] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:03.484297 3284 tablet_replica.cc:439] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } }
I20260924 23:16:03.484398 3284 tablet_replica.cc:442] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:16:03.484444 3281 tablet_replica.cc:439] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } }
I20260924 23:16:03.484534 3281 tablet_replica.cc:442] T 9b1481371c21426a9e054afa3624f1b8 P 306cc64cbde348e0b72e3c588e8561cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:03.485072 3299 mvcc.cc:205] Tried to move back new op lower bound from 7333035063218458624 to 7333035063153569792. Current Snapshot: MvccSnapshot[applied={T|T < 7333035063218458624 or (T in {7333035063218458624})}]
I20260924 23:16:03.485759 3282 tablet_replica.cc:439] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } }
I20260924 23:16:03.485836 3282 tablet_replica.cc:442] T 9b1481371c21426a9e054afa3624f1b8 P 9b9dec66e98846e78cd730fe92c8a6b7: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:16:03.486294 3280 tablet_replica.cc:439] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } } }
I20260924 23:16:03.486382 3280 tablet_replica.cc:442] T 9b1481371c21426a9e054afa3624f1b8 P 6c0d2232d9f840369602e1f768053fd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:03.486768 3301 mvcc.cc:205] Tried to move back new op lower bound from 7333035063218458624 to 7333035063153569792. Current Snapshot: MvccSnapshot[applied={T|T < 7333035063218458624}]
I20260924 23:16:03.489933 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.487478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.1.4.195:34167)
0924 23:16:03.487521 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.487523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.487569 (+ 46us) server_negotiation.cc:407] Connection header received
0924 23:16:03.487620 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.487621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.487630 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.487691 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:03.487770 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.488527 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.489281 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.489585 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.489645 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.489734 (+ 89us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:03.489764 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:03.489803 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.489843 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.489861 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:16:03.494894 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.492364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:44121 (local address 127.1.4.195:34167)
0924 23:16:03.492413 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.492414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.492422 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:03.492475 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.492475 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.492485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.492577 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:03.492661 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.493339 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.494101 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.494391 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.494448 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.494606 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:03.494638 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:03.494683 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.494796 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.494813 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 23:16:03.494987 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.492273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:34167 (local address 127.1.4.195:44121)
0924 23:16:03.492362 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:03.492373 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:03.492428 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:03.492576 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:03.492577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:03.492578 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:16:03.492623 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.492624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.493432 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.493433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.494035 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.494037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.494460 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.494461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.494520 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.494533 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:16:03.494729 (+ 196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:16:03.494732 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:03.494788 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 23:16:03.494825 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 23:16:03.497107 2925 raft_consensus.cc:1261] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3 [term 1 FOLLOWER]: Refusing update from remote peer 9b9dec66e98846e78cd730fe92c8a6b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:03.497148 3065 raft_consensus.cc:1261] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Refusing update from remote peer 9b9dec66e98846e78cd730fe92c8a6b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:03.497305 3310 consensus_queue.cc:1035] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:03.497395 3282 consensus_queue.cc:1035] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:03.499728 3300 mvcc.cc:205] Tried to move back new op lower bound from 7333035063282765824 to 7333035062706884608. Current Snapshot: MvccSnapshot[applied={T|T < 7333035063282765824}]
I20260924 23:16:03.512238 2767 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:03.512341 2767 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:03.513027 2767 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc: Sending DeleteTablet for 3 replicas of tablet df96de9e26d44728a4a68c3048065532
I20260924 23:16:03.513314 2905 tablet_service.cc:1514] Processing DeleteTablet for tablet df96de9e26d44728a4a68c3048065532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:03 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:35452
I20260924 23:16:03.513386 3040 tablet_service.cc:1514] Processing DeleteTablet for tablet df96de9e26d44728a4a68c3048065532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:03 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:54150
I20260924 23:16:03.513625 3316 tablet_replica.cc:330] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb: stopping tablet replica
I20260924 23:16:03.513628 3184 tablet_service.cc:1514] Processing DeleteTablet for tablet df96de9e26d44728a4a68c3048065532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:03 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:46652
I20260924 23:16:03.513748 3316 raft_consensus.cc:2229] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:03.513818 3316 raft_consensus.cc:2258] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:03.513826 2767 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:16:03.513962 2767 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:03.513993 3315 tablet_replica.cc:330] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3: stopping tablet replica
I20260924 23:16:03.514097 3315 raft_consensus.cc:2229] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:03.514145 3315 raft_consensus.cc:2258] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:03.514662 3316 ts_tablet_manager.cc:1905] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:03.515194 2905 tablet_service.cc:1467] Processing CreateTablet for tablet a4af193ac26f4fc3b3e358b0a3c75e93 (DEFAULT_TABLE table=test-table [id=f758a6baf437454f8f68bd4f71822284]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:03.515311 2905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4af193ac26f4fc3b3e358b0a3c75e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:03.515303 3040 tablet_service.cc:1467] Processing CreateTablet for tablet a4af193ac26f4fc3b3e358b0a3c75e93 (DEFAULT_TABLE table=test-table [id=f758a6baf437454f8f68bd4f71822284]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:03.515417 3040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4af193ac26f4fc3b3e358b0a3c75e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:03.516417 3316 ts_tablet_manager.cc:1918] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:16:03.516479 3316 log.cc:1196] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/df96de9e26d44728a4a68c3048065532
I20260924 23:16:03.516683 3275 tablet_bootstrap.cc:492] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3: Bootstrap starting.
I20260924 23:16:03.516710 3277 tablet_bootstrap.cc:492] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb: Bootstrap starting.
I20260924 23:16:03.516743 3316 ts_tablet_manager.cc:1939] T df96de9e26d44728a4a68c3048065532 P 306cc64cbde348e0b72e3c588e8561cb: Deleting consensus metadata
I20260924 23:16:03.517079 3275 tablet_bootstrap.cc:654] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:03.517099 3277 tablet_bootstrap.cc:654] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:03.517127 2754 catalog_manager.cc:4841] TS 306cc64cbde348e0b72e3c588e8561cb (127.1.4.194:38977): tablet df96de9e26d44728a4a68c3048065532 (table test-table [id=ab7a73783f504095bb6a6e179a21f850]) successfully deleted
I20260924 23:16:03.517210 3184 tablet_service.cc:1467] Processing CreateTablet for tablet a4af193ac26f4fc3b3e358b0a3c75e93 (DEFAULT_TABLE table=test-table [id=f758a6baf437454f8f68bd4f71822284]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:03.517263 3317 tablet_replica.cc:330] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7: stopping tablet replica
I20260924 23:16:03.517310 3184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4af193ac26f4fc3b3e358b0a3c75e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:03.517369 3317 raft_consensus.cc:2229] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:16:03.517465 3317 raft_consensus.cc:2258] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:03.517854 3275 tablet_bootstrap.cc:492] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3: No bootstrap required, opened a new log
I20260924 23:16:03.517906 3275 ts_tablet_manager.cc:1397] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:03.518030 3275 raft_consensus.cc:348] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.518102 3275 raft_consensus.cc:374] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:03.518121 3275 raft_consensus.cc:729] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0d2232d9f840369602e1f768053fd3, State: Initialized, Role: FOLLOWER
I20260924 23:16:03.518157 3275 consensus_queue.cc:260] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.518235 3275 ts_tablet_manager.cc:1428] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:03.518615 3274 tablet_bootstrap.cc:492] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7: Bootstrap starting.
I20260924 23:16:03.518694 3277 tablet_bootstrap.cc:492] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb: No bootstrap required, opened a new log
I20260924 23:16:03.518731 3277 ts_tablet_manager.cc:1397] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 23:16:03.518857 3277 raft_consensus.cc:348] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.518915 3277 raft_consensus.cc:374] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:03.518932 3277 raft_consensus.cc:729] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306cc64cbde348e0b72e3c588e8561cb, State: Initialized, Role: FOLLOWER
I20260924 23:16:03.518967 3277 consensus_queue.cc:260] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.518985 3274 tablet_bootstrap.cc:654] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:03.519057 3277 ts_tablet_manager.cc:1428] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:03.519318 3317 ts_tablet_manager.cc:1905] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:03.520498 3315 ts_tablet_manager.cc:1905] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:03.520663 3317 ts_tablet_manager.cc:1918] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:16:03.520730 3317 log.cc:1196] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/df96de9e26d44728a4a68c3048065532
I20260924 23:16:03.520963 3317 ts_tablet_manager.cc:1939] T df96de9e26d44728a4a68c3048065532 P 9b9dec66e98846e78cd730fe92c8a6b7: Deleting consensus metadata
I20260924 23:16:03.521376 2752 catalog_manager.cc:4841] TS 9b9dec66e98846e78cd730fe92c8a6b7 (127.1.4.195:34167): tablet df96de9e26d44728a4a68c3048065532 (table test-table [id=ab7a73783f504095bb6a6e179a21f850]) successfully deleted
I20260924 23:16:03.521776 3315 ts_tablet_manager.cc:1918] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:16:03.521837 3315 log.cc:1196] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/wal/wals/df96de9e26d44728a4a68c3048065532
I20260924 23:16:03.522042 3315 ts_tablet_manager.cc:1939] T df96de9e26d44728a4a68c3048065532 P 6c0d2232d9f840369602e1f768053fd3: Deleting consensus metadata
I20260924 23:16:03.522728 2753 catalog_manager.cc:4841] TS 6c0d2232d9f840369602e1f768053fd3 (127.1.4.193:44559): tablet df96de9e26d44728a4a68c3048065532 (table test-table [id=ab7a73783f504095bb6a6e179a21f850]) successfully deleted
I20260924 23:16:03.523778 3274 tablet_bootstrap.cc:492] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7: No bootstrap required, opened a new log
I20260924 23:16:03.523845 3274 ts_tablet_manager.cc:1397] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260924 23:16:03.523988 3274 raft_consensus.cc:348] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.524065 3274 raft_consensus.cc:374] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:03.524084 3274 raft_consensus.cc:729] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9dec66e98846e78cd730fe92c8a6b7, State: Initialized, Role: FOLLOWER
I20260924 23:16:03.524128 3274 consensus_queue.cc:260] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.524220 3274 ts_tablet_manager.cc:1428] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 23:16:03.896823 3280 raft_consensus.cc:482] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:03.896934 3280 raft_consensus.cc:504] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.897276 3280 leader_election.cc:288] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 306cc64cbde348e0b72e3c588e8561cb (127.1.4.194:38977), 9b9dec66e98846e78cd730fe92c8a6b7 (127.1.4.195:34167)
I20260924 23:16:03.900071 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.897577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:57043 (local address 127.1.4.195:34167)
0924 23:16:03.897627 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.897629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.897665 (+ 36us) server_negotiation.cc:407] Connection header received
0924 23:16:03.897742 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.897743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.897754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.897803 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:03.897922 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.898409 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.899410 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.899768 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.899818 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.899828 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.899964 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.900003 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 23:16:03.900092 3322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.897452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:34167 (local address 127.1.4.193:57043)
0924 23:16:03.897591 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:03.897603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:03.897669 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:03.897817 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:03.897818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:03.897819 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:03.897860 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.897862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.898488 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.898489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.899291 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.899293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.899840 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.899841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.899897 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.899901 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:03.899945 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:16:03.899986 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":31,"thread_start_us":69,"threads_started":1}
I20260924 23:16:03.900117 3323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.897546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:39183 (local address 127.1.4.194:38977)
0924 23:16:03.897657 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.897659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.897669 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:16:03.897722 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.897723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.897732 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.897787 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:03.897919 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.898399 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.899433 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.899813 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.899882 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.899893 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.900013 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.900055 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":52,"threads_started":1}
I20260924 23:16:03.900125 3321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.897456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:38977 (local address 127.1.4.193:39183)
0924 23:16:03.897595 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:03.897604 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:03.897661 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:03.897799 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:03.897800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:03.897802 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:03.897859 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.897861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.898471 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.898472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.899339 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.899341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.899898 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.899899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.899947 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.899951 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:03.900001 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 23:16:03.900038 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":38,"thread_start_us":70,"threads_started":1}
I20260924 23:16:03.900345 3065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4af193ac26f4fc3b3e358b0a3c75e93" candidate_uuid: "6c0d2232d9f840369602e1f768053fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306cc64cbde348e0b72e3c588e8561cb" is_pre_election: true
I20260924 23:16:03.900449 3065 raft_consensus.cc:2454] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c0d2232d9f840369602e1f768053fd3 in term 0.
I20260924 23:16:03.900490 3197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4af193ac26f4fc3b3e358b0a3c75e93" candidate_uuid: "6c0d2232d9f840369602e1f768053fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" is_pre_election: true
I20260924 23:16:03.900619 3197 raft_consensus.cc:2454] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c0d2232d9f840369602e1f768053fd3 in term 0.
I20260924 23:16:03.900641 2858 leader_election.cc:302] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306cc64cbde348e0b72e3c588e8561cb, 6c0d2232d9f840369602e1f768053fd3; no voters:
I20260924 23:16:03.900780 3280 raft_consensus.cc:2781] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:03.900828 3280 raft_consensus.cc:482] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:03.900849 3280 raft_consensus.cc:3037] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:03.901532 3280 raft_consensus.cc:504] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.901661 3280 leader_election.cc:288] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [CANDIDATE]: Term 1 election: Requested vote from peers 306cc64cbde348e0b72e3c588e8561cb (127.1.4.194:38977), 9b9dec66e98846e78cd730fe92c8a6b7 (127.1.4.195:34167)
I20260924 23:16:03.901805 3197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4af193ac26f4fc3b3e358b0a3c75e93" candidate_uuid: "6c0d2232d9f840369602e1f768053fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9dec66e98846e78cd730fe92c8a6b7"
I20260924 23:16:03.901875 3197 raft_consensus.cc:3037] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:03.901861 3065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4af193ac26f4fc3b3e358b0a3c75e93" candidate_uuid: "6c0d2232d9f840369602e1f768053fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306cc64cbde348e0b72e3c588e8561cb"
I20260924 23:16:03.901939 3065 raft_consensus.cc:3037] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:03.902558 3197 raft_consensus.cc:2454] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c0d2232d9f840369602e1f768053fd3 in term 1.
I20260924 23:16:03.902577 3065 raft_consensus.cc:2454] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c0d2232d9f840369602e1f768053fd3 in term 1.
I20260924 23:16:03.902738 2858 leader_election.cc:302] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306cc64cbde348e0b72e3c588e8561cb, 6c0d2232d9f840369602e1f768053fd3; no voters:
I20260924 23:16:03.902840 3280 raft_consensus.cc:2781] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:03.902974 3280 raft_consensus.cc:686] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [term 1 LEADER]: Becoming Leader. State: Replica: 6c0d2232d9f840369602e1f768053fd3, State: Running, Role: LEADER
I20260924 23:16:03.903038 3280 consensus_queue.cc:237] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:03.903735 2768 catalog_manager.cc:5495] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c0d2232d9f840369602e1f768053fd3 (127.1.4.193). New cstate: current_term: 1 leader_uuid: "6c0d2232d9f840369602e1f768053fd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } health_report { overall_health: UNKNOWN } } }
I20260924 23:16:03.961256 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.958870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.1.4.193:44559)
0924 23:16:03.958981 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.958983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.958989 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:16:03.959021 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.959022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.959048 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.959115 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:03.959193 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.959705 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.960529 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.960739 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.960792 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.960959 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:03.961071 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:03.961122 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.961159 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.961176 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
I20260924 23:16:03.965939 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.963828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:41157 (local address 127.1.4.195:34167)
0924 23:16:03.963877 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.963879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.963969 (+ 90us) server_negotiation.cc:407] Connection header received
0924 23:16:03.964010 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.964011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.964019 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.964073 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:03.964242 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.964683 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.965408 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.965601 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.965644 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.965776 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:03.965799 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:03.965829 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.965868 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.965881 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 23:16:03.966114 3329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.963760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:34167 (local address 127.1.4.193:41157)
0924 23:16:03.963910 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:03.963920 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:03.963972 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:03.964113 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:03.964115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:03.964117 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:16:03.964175 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.964177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.964759 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.964760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.965349 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.965350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.965667 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.965668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.965709 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.965721 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:16:03.965837 (+ 116us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:16:03.965838 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 23:16:03.965867 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 23:16:03.965899 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260924 23:16:03.968379 3065 raft_consensus.cc:1261] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Refusing update from remote peer 6c0d2232d9f840369602e1f768053fd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:03.968492 3197 raft_consensus.cc:1261] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Refusing update from remote peer 6c0d2232d9f840369602e1f768053fd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:03.968744 3324 consensus_queue.cc:1035] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:03.968858 3280 consensus_queue.cc:1035] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:03.975721 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.973728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:39139 (local address 127.1.4.193:44559)
0924 23:16:03.973764 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:03.973765 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:03.973770 (+ 5us) server_negotiation.cc:407] Connection header received
0924 23:16:03.973787 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:03.973788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:03.973796 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:03.973828 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:03.973943 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.974410 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.975174 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.975398 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.975460 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.975548 (+ 88us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:03.975575 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:03.975606 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:03.975638 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 23:16:03.975653 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:16:03.975736 3257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:03.973581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:44559 (local address 127.1.4.195:39139)
0924 23:16:03.973622 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:03.973634 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:03.973671 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:03.973843 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:03.973844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:03.973846 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:16:03.973892 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.973894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.974496 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.974497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.975103 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:03.975106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:03.975456 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:03.975457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:03.975497 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:03.975502 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:16:03.975606 (+ 104us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:16:03.975607 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 23:16:03.975634 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 23:16:03.975669 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 23:16:03.976090 3300 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 23:16:05.495853 2768 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:05.495951 2768 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:05.496865 2768 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc: Sending DeleteTablet for 3 replicas of tablet a4af193ac26f4fc3b3e358b0a3c75e93
I20260924 23:16:05.497140 2905 tablet_service.cc:1514] Processing DeleteTablet for tablet a4af193ac26f4fc3b3e358b0a3c75e93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:05 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:35452
I20260924 23:16:05.497177 3040 tablet_service.cc:1514] Processing DeleteTablet for tablet a4af193ac26f4fc3b3e358b0a3c75e93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:05 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:54150
I20260924 23:16:05.497176 3184 tablet_service.cc:1514] Processing DeleteTablet for tablet a4af193ac26f4fc3b3e358b0a3c75e93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:05 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:46652
I20260924 23:16:05.497315 3357 tablet_replica.cc:330] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3: stopping tablet replica
I20260924 23:16:05.497366 3357 raft_consensus.cc:2229] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:16:05.497371 3358 tablet_replica.cc:330] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7: stopping tablet replica
I20260924 23:16:05.497414 3358 raft_consensus.cc:2229] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:05.497443 3358 raft_consensus.cc:2258] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:05.497453 3357 raft_consensus.cc:2258] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:05.497689 3358 ts_tablet_manager.cc:1905] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:05.497723 3357 ts_tablet_manager.cc:1905] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:05.499509 3357 ts_tablet_manager.cc:1918] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 23:16:05.499562 3357 log.cc:1196] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/wal/wals/a4af193ac26f4fc3b3e358b0a3c75e93
I20260924 23:16:05.499517 3358 ts_tablet_manager.cc:1918] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 23:16:05.499598 3358 log.cc:1196] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/a4af193ac26f4fc3b3e358b0a3c75e93
I20260924 23:16:05.499766 3357 ts_tablet_manager.cc:1939] T a4af193ac26f4fc3b3e358b0a3c75e93 P 6c0d2232d9f840369602e1f768053fd3: Deleting consensus metadata
I20260924 23:16:05.499781 3358 ts_tablet_manager.cc:1939] T a4af193ac26f4fc3b3e358b0a3c75e93 P 9b9dec66e98846e78cd730fe92c8a6b7: Deleting consensus metadata
I20260924 23:16:05.500044 2752 catalog_manager.cc:4841] TS 9b9dec66e98846e78cd730fe92c8a6b7 (127.1.4.195:34167): tablet a4af193ac26f4fc3b3e358b0a3c75e93 (table test-table [id=f758a6baf437454f8f68bd4f71822284]) successfully deleted
I20260924 23:16:05.500054 2753 catalog_manager.cc:4841] TS 6c0d2232d9f840369602e1f768053fd3 (127.1.4.193:44559): tablet a4af193ac26f4fc3b3e358b0a3c75e93 (table test-table [id=f758a6baf437454f8f68bd4f71822284]) successfully deleted
I20260924 23:16:05.501746 3359 tablet_replica.cc:330] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb: stopping tablet replica
I20260924 23:16:05.501806 3359 raft_consensus.cc:2229] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:05.501848 3359 raft_consensus.cc:2258] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:05.502897 3359 ts_tablet_manager.cc:1905] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:05.504724 3359 ts_tablet_manager.cc:1918] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 23:16:05.504793 3359 log.cc:1196] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/a4af193ac26f4fc3b3e358b0a3c75e93
I20260924 23:16:05.505002 3359 ts_tablet_manager.cc:1939] T a4af193ac26f4fc3b3e358b0a3c75e93 P 306cc64cbde348e0b72e3c588e8561cb: Deleting consensus metadata
I20260924 23:16:05.505442 2754 catalog_manager.cc:4841] TS 306cc64cbde348e0b72e3c588e8561cb (127.1.4.194:38977): tablet a4af193ac26f4fc3b3e358b0a3c75e93 (table test-table [id=f758a6baf437454f8f68bd4f71822284]) successfully deleted
Sep 24 23:16:05 dist-test-slave-s1hw krb5kdc[2723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291765, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 23:16:05 dist-test-slave-s1hw krb5kdc[2723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291765, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:05.526507 3367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:05.513246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58876 (local address 127.1.4.254:38247)
0924 23:16:05.513366 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:05.513368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:05.513394 (+ 26us) server_negotiation.cc:407] Connection header received
0924 23:16:05.513468 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:05.513469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:05.513478 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:05.513548 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:05.514138 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.514687 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.515669 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.515877 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.516007 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:05.523977 (+ 7970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:05.523979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:05.523979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:05.523989 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:05.525332 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:05.525690 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:05.525692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:05.525692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:05.525763 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:05.525962 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:05.525963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:05.525963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:05.526095 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:05.526187 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:05.526335 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 23:16:05.526421 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
I20260924 23:16:05.528066 2768 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:16:05.528210 2768 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:05.529675 2905 tablet_service.cc:1467] Processing CreateTablet for tablet 6166e83be71e440fbb0d0cc28b5d5469 (DEFAULT_TABLE table=test-table [id=b832bab7ca1047ed95fe7d52b65aaf70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:05.529691 3040 tablet_service.cc:1467] Processing CreateTablet for tablet 6166e83be71e440fbb0d0cc28b5d5469 (DEFAULT_TABLE table=test-table [id=b832bab7ca1047ed95fe7d52b65aaf70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:05.529731 3184 tablet_service.cc:1467] Processing CreateTablet for tablet 6166e83be71e440fbb0d0cc28b5d5469 (DEFAULT_TABLE table=test-table [id=b832bab7ca1047ed95fe7d52b65aaf70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:05.529824 3184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6166e83be71e440fbb0d0cc28b5d5469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:05.529824 3040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6166e83be71e440fbb0d0cc28b5d5469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:05.529824 2905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6166e83be71e440fbb0d0cc28b5d5469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:05.531121 3368 tablet_bootstrap.cc:492] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3: Bootstrap starting.
I20260924 23:16:05.531276 3370 tablet_bootstrap.cc:492] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7: Bootstrap starting.
I20260924 23:16:05.531500 3368 tablet_bootstrap.cc:654] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:05.531667 3370 tablet_bootstrap.cc:654] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:05.531757 3369 tablet_bootstrap.cc:492] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb: Bootstrap starting.
I20260924 23:16:05.532002 3368 tablet_bootstrap.cc:492] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3: No bootstrap required, opened a new log
I20260924 23:16:05.532045 3368 ts_tablet_manager.cc:1397] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 23:16:05.532145 3368 raft_consensus.cc:348] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.532212 3370 tablet_bootstrap.cc:492] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7: No bootstrap required, opened a new log
I20260924 23:16:05.532220 3368 raft_consensus.cc:374] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:05.532245 3368 raft_consensus.cc:729] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0d2232d9f840369602e1f768053fd3, State: Initialized, Role: FOLLOWER
I20260924 23:16:05.532258 3370 ts_tablet_manager.cc:1397] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:05.532294 3368 consensus_queue.cc:260] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.532400 3368 ts_tablet_manager.cc:1428] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:05.532372 3370 raft_consensus.cc:348] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.532439 3370 raft_consensus.cc:374] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:05.532465 3370 raft_consensus.cc:729] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9dec66e98846e78cd730fe92c8a6b7, State: Initialized, Role: FOLLOWER
I20260924 23:16:05.532529 3370 consensus_queue.cc:260] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.532651 3370 ts_tablet_manager.cc:1428] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:05.532737 3369 tablet_bootstrap.cc:654] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:05.533303 3369 tablet_bootstrap.cc:492] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb: No bootstrap required, opened a new log
I20260924 23:16:05.533353 3369 ts_tablet_manager.cc:1397] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:05.533438 3369 raft_consensus.cc:348] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.533490 3369 raft_consensus.cc:374] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:05.533501 3369 raft_consensus.cc:729] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306cc64cbde348e0b72e3c588e8561cb, State: Initialized, Role: FOLLOWER
I20260924 23:16:05.533537 3369 consensus_queue.cc:260] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.533681 3369 ts_tablet_manager.cc:1428] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:05.547580 3347 raft_consensus.cc:482] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:05.547667 3347 raft_consensus.cc:504] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.547796 3347 leader_election.cc:288] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 306cc64cbde348e0b72e3c588e8561cb (127.1.4.194:38977), 9b9dec66e98846e78cd730fe92c8a6b7 (127.1.4.195:34167)
I20260924 23:16:05.547996 3197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6166e83be71e440fbb0d0cc28b5d5469" candidate_uuid: "6c0d2232d9f840369602e1f768053fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" is_pre_election: true
I20260924 23:16:05.547993 3065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6166e83be71e440fbb0d0cc28b5d5469" candidate_uuid: "6c0d2232d9f840369602e1f768053fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306cc64cbde348e0b72e3c588e8561cb" is_pre_election: true
I20260924 23:16:05.548082 3197 raft_consensus.cc:2454] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c0d2232d9f840369602e1f768053fd3 in term 0.
I20260924 23:16:05.548092 3065 raft_consensus.cc:2454] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c0d2232d9f840369602e1f768053fd3 in term 0.
I20260924 23:16:05.548235 2858 leader_election.cc:302] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306cc64cbde348e0b72e3c588e8561cb, 6c0d2232d9f840369602e1f768053fd3; no voters:
I20260924 23:16:05.548321 3347 raft_consensus.cc:2781] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:05.548372 3347 raft_consensus.cc:482] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:05.548396 3347 raft_consensus.cc:3037] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:05.549062 3347 raft_consensus.cc:504] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.549180 3347 leader_election.cc:288] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [CANDIDATE]: Term 1 election: Requested vote from peers 306cc64cbde348e0b72e3c588e8561cb (127.1.4.194:38977), 9b9dec66e98846e78cd730fe92c8a6b7 (127.1.4.195:34167)
I20260924 23:16:05.549322 3197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6166e83be71e440fbb0d0cc28b5d5469" candidate_uuid: "6c0d2232d9f840369602e1f768053fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9dec66e98846e78cd730fe92c8a6b7"
I20260924 23:16:05.549324 3065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6166e83be71e440fbb0d0cc28b5d5469" candidate_uuid: "6c0d2232d9f840369602e1f768053fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "306cc64cbde348e0b72e3c588e8561cb"
I20260924 23:16:05.549393 3065 raft_consensus.cc:3037] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:05.549393 3197 raft_consensus.cc:3037] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:05.550098 3197 raft_consensus.cc:2454] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c0d2232d9f840369602e1f768053fd3 in term 1.
I20260924 23:16:05.550098 3065 raft_consensus.cc:2454] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c0d2232d9f840369602e1f768053fd3 in term 1.
I20260924 23:16:05.550271 2856 leader_election.cc:302] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c0d2232d9f840369602e1f768053fd3, 9b9dec66e98846e78cd730fe92c8a6b7; no voters:
I20260924 23:16:05.550371 3347 raft_consensus.cc:2781] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:05.550464 3347 raft_consensus.cc:686] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [term 1 LEADER]: Becoming Leader. State: Replica: 6c0d2232d9f840369602e1f768053fd3, State: Running, Role: LEADER
I20260924 23:16:05.550514 3347 consensus_queue.cc:237] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.551091 2766 catalog_manager.cc:5495] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c0d2232d9f840369602e1f768053fd3 (127.1.4.193). New cstate: current_term: 1 leader_uuid: "6c0d2232d9f840369602e1f768053fd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } health_report { overall_health: UNKNOWN } } }
I20260924 23:16:05.563700 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:05.561595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.1.4.193:44559)
0924 23:16:05.561761 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:05.561762 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:05.561769 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:16:05.561788 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:05.561788 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:05.561798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:05.561854 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:05.561993 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.562460 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.563183 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.563385 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.563429 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:05.563528 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:05.563552 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:05.563582 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:05.563613 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 23:16:05.563629 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
I20260924 23:16:05.566416 3197 raft_consensus.cc:1261] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Refusing update from remote peer 6c0d2232d9f840369602e1f768053fd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:05.566547 3065 raft_consensus.cc:1261] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Refusing update from remote peer 6c0d2232d9f840369602e1f768053fd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:05.566627 3280 consensus_queue.cc:1035] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:05.566874 3280 consensus_queue.cc:1035] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:05.576100 2766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:05.576195 2766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:05.576725 2766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc: Sending DeleteTablet for 3 replicas of tablet 6166e83be71e440fbb0d0cc28b5d5469
I20260924 23:16:05.576942 3040 tablet_service.cc:1514] Processing DeleteTablet for tablet 6166e83be71e440fbb0d0cc28b5d5469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:05 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:54150
I20260924 23:16:05.576951 2905 tablet_service.cc:1514] Processing DeleteTablet for tablet 6166e83be71e440fbb0d0cc28b5d5469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:05 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:35452
I20260924 23:16:05.576947 3184 tablet_service.cc:1514] Processing DeleteTablet for tablet 6166e83be71e440fbb0d0cc28b5d5469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:05 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:46652
I20260924 23:16:05.577034 3359 tablet_replica.cc:330] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb: stopping tablet replica
I20260924 23:16:05.577036 3358 tablet_replica.cc:330] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7: stopping tablet replica
I20260924 23:16:05.577080 3359 raft_consensus.cc:2229] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:05.577088 3357 tablet_replica.cc:330] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3: stopping tablet replica
I20260924 23:16:05.577087 3358 raft_consensus.cc:2229] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:05.577116 3359 raft_consensus.cc:2258] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:05.577126 3358 raft_consensus.cc:2258] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:05.577136 3357 raft_consensus.cc:2229] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:16:05.577183 3357 raft_consensus.cc:2258] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:05.577448 3357 ts_tablet_manager.cc:1905] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:05.577461 3358 ts_tablet_manager.cc:1905] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:05.577510 3359 ts_tablet_manager.cc:1905] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:05.577476 2766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:16:05.577602 2766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:05.578716 2905 tablet_service.cc:1467] Processing CreateTablet for tablet 14160b534d1c4a48a5c96a09b40f9bc8 (DEFAULT_TABLE table=test-table [id=912b4363c0a14e8b89e424d548dee139]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:05.578930 3184 tablet_service.cc:1467] Processing CreateTablet for tablet 14160b534d1c4a48a5c96a09b40f9bc8 (DEFAULT_TABLE table=test-table [id=912b4363c0a14e8b89e424d548dee139]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:05.579052 3040 tablet_service.cc:1467] Processing CreateTablet for tablet 14160b534d1c4a48a5c96a09b40f9bc8 (DEFAULT_TABLE table=test-table [id=912b4363c0a14e8b89e424d548dee139]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:05.579166 3040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14160b534d1c4a48a5c96a09b40f9bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:05.579173 3359 ts_tablet_manager.cc:1918] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:16:05.579252 3359 log.cc:1196] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/6166e83be71e440fbb0d0cc28b5d5469
I20260924 23:16:05.579351 3357 ts_tablet_manager.cc:1918] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:16:05.579444 3357 log.cc:1196] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-0/wal/wals/6166e83be71e440fbb0d0cc28b5d5469
I20260924 23:16:05.579483 3359 ts_tablet_manager.cc:1939] T 6166e83be71e440fbb0d0cc28b5d5469 P 306cc64cbde348e0b72e3c588e8561cb: Deleting consensus metadata
I20260924 23:16:05.579640 3357 ts_tablet_manager.cc:1939] T 6166e83be71e440fbb0d0cc28b5d5469 P 6c0d2232d9f840369602e1f768053fd3: Deleting consensus metadata
I20260924 23:16:05.579797 3358 ts_tablet_manager.cc:1918] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:16:05.579792 2754 catalog_manager.cc:4841] TS 306cc64cbde348e0b72e3c588e8561cb (127.1.4.194:38977): tablet 6166e83be71e440fbb0d0cc28b5d5469 (table test-table [id=b832bab7ca1047ed95fe7d52b65aaf70]) successfully deleted
I20260924 23:16:05.579813 3184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14160b534d1c4a48a5c96a09b40f9bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:05.579855 3358 log.cc:1196] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/6166e83be71e440fbb0d0cc28b5d5469
I20260924 23:16:05.580034 3358 ts_tablet_manager.cc:1939] T 6166e83be71e440fbb0d0cc28b5d5469 P 9b9dec66e98846e78cd730fe92c8a6b7: Deleting consensus metadata
I20260924 23:16:05.580264 3369 tablet_bootstrap.cc:492] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb: Bootstrap starting.
I20260924 23:16:05.580291 2905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14160b534d1c4a48a5c96a09b40f9bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:05.580322 2752 catalog_manager.cc:4841] TS 9b9dec66e98846e78cd730fe92c8a6b7 (127.1.4.195:34167): tablet 6166e83be71e440fbb0d0cc28b5d5469 (table test-table [id=b832bab7ca1047ed95fe7d52b65aaf70]) successfully deleted
I20260924 23:16:05.580564 3369 tablet_bootstrap.cc:654] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:05.580880 3370 tablet_bootstrap.cc:492] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7: Bootstrap starting.
I20260924 23:16:05.581169 3370 tablet_bootstrap.cc:654] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:05.581435 3368 tablet_bootstrap.cc:492] T 14160b534d1c4a48a5c96a09b40f9bc8 P 6c0d2232d9f840369602e1f768053fd3: Bootstrap starting.
I20260924 23:16:05.581626 3370 tablet_bootstrap.cc:492] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7: No bootstrap required, opened a new log
I20260924 23:16:05.581670 3370 ts_tablet_manager.cc:1397] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 23:16:05.581677 3368 tablet_bootstrap.cc:654] T 14160b534d1c4a48a5c96a09b40f9bc8 P 6c0d2232d9f840369602e1f768053fd3: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:05.581774 3370 raft_consensus.cc:348] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.581827 3370 raft_consensus.cc:374] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:05.581838 3370 raft_consensus.cc:729] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9dec66e98846e78cd730fe92c8a6b7, State: Initialized, Role: FOLLOWER
I20260924 23:16:05.581871 3370 consensus_queue.cc:260] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.581959 3370 ts_tablet_manager.cc:1428] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:05.582231 2753 catalog_manager.cc:4841] TS 6c0d2232d9f840369602e1f768053fd3 (127.1.4.193:44559): tablet 6166e83be71e440fbb0d0cc28b5d5469 (table test-table [id=b832bab7ca1047ed95fe7d52b65aaf70]) successfully deleted
I20260924 23:16:05.582437 3368 tablet_bootstrap.cc:492] T 14160b534d1c4a48a5c96a09b40f9bc8 P 6c0d2232d9f840369602e1f768053fd3: No bootstrap required, opened a new log
I20260924 23:16:05.582489 3368 ts_tablet_manager.cc:1397] T 14160b534d1c4a48a5c96a09b40f9bc8 P 6c0d2232d9f840369602e1f768053fd3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 23:16:05.582582 3369 tablet_bootstrap.cc:492] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb: No bootstrap required, opened a new log
I20260924 23:16:05.582587 3368 raft_consensus.cc:348] T 14160b534d1c4a48a5c96a09b40f9bc8 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.582633 3369 ts_tablet_manager.cc:1397] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:05.582656 3368 raft_consensus.cc:374] T 14160b534d1c4a48a5c96a09b40f9bc8 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:05.582683 3368 raft_consensus.cc:729] T 14160b534d1c4a48a5c96a09b40f9bc8 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0d2232d9f840369602e1f768053fd3, State: Initialized, Role: FOLLOWER
I20260924 23:16:05.582717 3368 consensus_queue.cc:260] T 14160b534d1c4a48a5c96a09b40f9bc8 P 6c0d2232d9f840369602e1f768053fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.582762 3369 raft_consensus.cc:348] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.582808 3368 ts_tablet_manager.cc:1428] T 14160b534d1c4a48a5c96a09b40f9bc8 P 6c0d2232d9f840369602e1f768053fd3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:05.582823 3369 raft_consensus.cc:374] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:05.582849 3369 raft_consensus.cc:729] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306cc64cbde348e0b72e3c588e8561cb, State: Initialized, Role: FOLLOWER
I20260924 23:16:05.582885 3369 consensus_queue.cc:260] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.582980 3369 ts_tablet_manager.cc:1428] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:05.619861 3374 raft_consensus.cc:482] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:05.619969 3374 raft_consensus.cc:504] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.620294 3374 leader_election.cc:288] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c0d2232d9f840369602e1f768053fd3 (127.1.4.193:44559), 9b9dec66e98846e78cd730fe92c8a6b7 (127.1.4.195:34167)
I20260924 23:16:05.623143 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:05.620528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:59199 (local address 127.1.4.193:44559)
0924 23:16:05.620616 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:05.620619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:05.620703 (+ 84us) server_negotiation.cc:407] Connection header received
0924 23:16:05.620763 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:05.620764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:05.620772 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:05.620819 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:05.620980 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.621464 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.622471 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.622833 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.622891 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:05.622905 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:05.623037 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 23:16:05.623072 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 23:16:05.623157 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:05.620564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:45779 (local address 127.1.4.195:34167)
0924 23:16:05.620675 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:05.620677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:05.620686 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:05.620739 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:05.620739 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:05.620750 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:05.620809 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:05.620959 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.621458 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.622476 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.622848 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.622906 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:05.622918 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:05.623041 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 23:16:05.623081 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260924 23:16:05.623193 3385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:05.620478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:44559 (local address 127.1.4.194:59199)
0924 23:16:05.620616 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:05.620629 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:05.620698 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:05.620825 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:05.620826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:05.620828 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:05.620925 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:05.620927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.621546 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.621547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:05.622395 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:05.622397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.622907 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.622907 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:05.622979 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:05.622983 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:05.623026 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 23:16:05.623069 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":22,"thread_start_us":71,"threads_started":1}
I20260924 23:16:05.623201 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:05.620486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:34167 (local address 127.1.4.194:45779)
0924 23:16:05.620623 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:05.620632 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:05.620689 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:05.620817 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:05.620818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:05.620820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:05.620899 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:05.620901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.621527 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.621528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:05.622386 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:05.622389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.622913 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.622914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:05.622981 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:05.622984 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:05.623027 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 23:16:05.623071 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":38,"thread_start_us":73,"threads_started":1}
I20260924 23:16:05.623574 2925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14160b534d1c4a48a5c96a09b40f9bc8" candidate_uuid: "306cc64cbde348e0b72e3c588e8561cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0d2232d9f840369602e1f768053fd3" is_pre_election: true
I20260924 23:16:05.623667 2925 raft_consensus.cc:2454] T 14160b534d1c4a48a5c96a09b40f9bc8 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 306cc64cbde348e0b72e3c588e8561cb in term 0.
I20260924 23:16:05.623675 3197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14160b534d1c4a48a5c96a09b40f9bc8" candidate_uuid: "306cc64cbde348e0b72e3c588e8561cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" is_pre_election: true
I20260924 23:16:05.623768 3197 raft_consensus.cc:2454] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 306cc64cbde348e0b72e3c588e8561cb in term 0.
I20260924 23:16:05.623833 2997 leader_election.cc:302] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306cc64cbde348e0b72e3c588e8561cb, 6c0d2232d9f840369602e1f768053fd3; no voters:
I20260924 23:16:05.623963 3374 raft_consensus.cc:2781] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:05.624002 3374 raft_consensus.cc:482] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:05.624028 3374 raft_consensus.cc:3037] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:05.624684 3374 raft_consensus.cc:504] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.624796 3374 leader_election.cc:288] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [CANDIDATE]: Term 1 election: Requested vote from peers 6c0d2232d9f840369602e1f768053fd3 (127.1.4.193:44559), 9b9dec66e98846e78cd730fe92c8a6b7 (127.1.4.195:34167)
I20260924 23:16:05.624948 2925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14160b534d1c4a48a5c96a09b40f9bc8" candidate_uuid: "306cc64cbde348e0b72e3c588e8561cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0d2232d9f840369602e1f768053fd3"
I20260924 23:16:05.625021 2925 raft_consensus.cc:3037] T 14160b534d1c4a48a5c96a09b40f9bc8 P 6c0d2232d9f840369602e1f768053fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:05.625010 3197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14160b534d1c4a48a5c96a09b40f9bc8" candidate_uuid: "306cc64cbde348e0b72e3c588e8561cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9dec66e98846e78cd730fe92c8a6b7"
I20260924 23:16:05.625095 3197 raft_consensus.cc:3037] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:05.625694 2925 raft_consensus.cc:2454] T 14160b534d1c4a48a5c96a09b40f9bc8 P 6c0d2232d9f840369602e1f768053fd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 306cc64cbde348e0b72e3c588e8561cb in term 1.
I20260924 23:16:05.625767 3197 raft_consensus.cc:2454] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 306cc64cbde348e0b72e3c588e8561cb in term 1.
I20260924 23:16:05.625824 2997 leader_election.cc:302] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 306cc64cbde348e0b72e3c588e8561cb, 6c0d2232d9f840369602e1f768053fd3; no voters:
I20260924 23:16:05.625901 3374 raft_consensus.cc:2781] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:05.626047 3374 raft_consensus.cc:686] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [term 1 LEADER]: Becoming Leader. State: Replica: 306cc64cbde348e0b72e3c588e8561cb, State: Running, Role: LEADER
I20260924 23:16:05.626109 3374 consensus_queue.cc:237] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } }
I20260924 23:16:05.626734 2768 catalog_manager.cc:5495] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 306cc64cbde348e0b72e3c588e8561cb (127.1.4.194). New cstate: current_term: 1 leader_uuid: "306cc64cbde348e0b72e3c588e8561cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "306cc64cbde348e0b72e3c588e8561cb" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 38977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 } health_report { overall_health: UNKNOWN } } }
I20260924 23:16:05.643115 3389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:05.640751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.1.4.194:38977)
0924 23:16:05.640916 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:05.640917 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:05.640923 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:16:05.640954 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:05.640954 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:05.640973 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:05.641070 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:05.641174 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.641711 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.642381 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.642602 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.642655 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:05.642788 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:05.642897 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:05.642960 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:05.642996 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 23:16:05.643015 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
I20260924 23:16:05.647322 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:05.645458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:43615 (local address 127.1.4.195:34167)
0924 23:16:05.645508 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:05.645510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:05.645519 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:05.645555 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:05.645556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:05.645566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:05.645606 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:05.645692 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.646146 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.646813 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.646991 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.647032 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:05.647162 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:05.647185 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:05.647216 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:05.647252 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 23:16:05.647268 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:16:05.647344 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:05.645321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:34167 (local address 127.1.4.194:43615)
0924 23:16:05.645463 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:05.645472 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:05.645513 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:05.645609 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:05.645609 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:05.645611 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:16:05.645658 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:05.645660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.646202 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.646203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:05.646756 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:05.646758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.647054 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.647055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:05.647097 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:05.647109 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:16:05.647221 (+ 112us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:16:05.647222 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 23:16:05.647250 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 23:16:05.647283 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":56,"threads_started":1}
I20260924 23:16:05.649212 2925 raft_consensus.cc:1261] T 14160b534d1c4a48a5c96a09b40f9bc8 P 6c0d2232d9f840369602e1f768053fd3 [term 1 FOLLOWER]: Refusing update from remote peer 306cc64cbde348e0b72e3c588e8561cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:05.649211 3197 raft_consensus.cc:1261] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Refusing update from remote peer 306cc64cbde348e0b72e3c588e8561cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:05.649434 3387 consensus_queue.cc:1035] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0d2232d9f840369602e1f768053fd3" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:05.649565 3374 consensus_queue.cc:1035] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b9dec66e98846e78cd730fe92c8a6b7" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 34167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:05.656232 3389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:05.653902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:50123 (local address 127.1.4.194:38977)
0924 23:16:05.653937 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:05.653939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:05.654042 (+ 103us) server_negotiation.cc:407] Connection header received
0924 23:16:05.654086 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:05.654086 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:05.654097 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:05.654148 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:05.654258 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.654719 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.655551 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.655840 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.655904 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:05.656023 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:05.656049 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:05.656081 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:05.656142 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 23:16:05.656160 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:16:05.656265 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:05.653845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:38977 (local address 127.1.4.195:50123)
0924 23:16:05.653991 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:05.654000 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:05.654048 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:05.654150 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:05.654151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:05.654152 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:16:05.654190 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:05.654191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.654778 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.654779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:05.655481 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:05.655484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:05.655902 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:05.655903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:05.655962 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:05.655968 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:16:05.656085 (+ 117us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:16:05.656086 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 23:16:05.656139 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 23:16:05.656184 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
W20260924 23:16:05.656558 3339 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 23:16:07.175184 2768 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:07.175279 2768 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:07.176009 2768 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2c224c3dedb4eab8516e3d1c342e3dc: Sending DeleteTablet for 3 replicas of tablet 14160b534d1c4a48a5c96a09b40f9bc8
I20260924 23:16:07.176232 3040 tablet_service.cc:1514] Processing DeleteTablet for tablet 14160b534d1c4a48a5c96a09b40f9bc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:07 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:54150
I20260924 23:16:07.176246 3184 tablet_service.cc:1514] Processing DeleteTablet for tablet 14160b534d1c4a48a5c96a09b40f9bc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:07 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:46652
I20260924 23:16:07.176415 3421 tablet_replica.cc:330] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb: stopping tablet replica
I20260924 23:16:07.176414 2905 tablet_service.cc:1514] Processing DeleteTablet for tablet 14160b534d1c4a48a5c96a09b40f9bc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:07 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:35452
I20260924 23:16:07.176476 3421 raft_consensus.cc:2229] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:16:07.176518 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 2840
I20260924 23:16:07.176999 3421 raft_consensus.cc:2258] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:07.177397 3421 ts_tablet_manager.cc:1905] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:07.179028 3421 ts_tablet_manager.cc:1918] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 23:16:07.179066 3421 log.cc:1196] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/14160b534d1c4a48a5c96a09b40f9bc8
I20260924 23:16:07.179211 3421 ts_tablet_manager.cc:1939] T 14160b534d1c4a48a5c96a09b40f9bc8 P 306cc64cbde348e0b72e3c588e8561cb: Deleting consensus metadata
I20260924 23:16:07.179464 2754 catalog_manager.cc:4841] TS 306cc64cbde348e0b72e3c588e8561cb (127.1.4.194:38977): tablet 14160b534d1c4a48a5c96a09b40f9bc8 (table test-table [id=912b4363c0a14e8b89e424d548dee139]) successfully deleted
W20260924 23:16:07.181775 2753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.4.193:44559 (error 108)
I20260924 23:16:07.181960 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 2980
I20260924 23:16:07.182813 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:07.182605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:44559 (local address 127.0.0.1:35476)
0924 23:16:07.182727 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:07.182774 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
W20260924 23:16:07.182948 2753 catalog_manager.cc:4584] TS 6c0d2232d9f840369602e1f768053fd3 (127.1.4.193:44559): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14160b534d1c4a48a5c96a09b40f9bc8: Network error: Client connection negotiation failed: client connection to 127.1.4.193:44559: connect: Connection refused (error 111)
I20260924 23:16:07.183827 3422 tablet_replica.cc:330] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7: stopping tablet replica
I20260924 23:16:07.183893 3422 raft_consensus.cc:2229] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:07.183944 3422 raft_consensus.cc:2258] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:07.184201 3422 ts_tablet_manager.cc:1905] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:07.185819 3422 ts_tablet_manager.cc:1918] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 23:16:07.185874 3422 log.cc:1196] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/14160b534d1c4a48a5c96a09b40f9bc8
I20260924 23:16:07.186065 3422 ts_tablet_manager.cc:1939] T 14160b534d1c4a48a5c96a09b40f9bc8 P 9b9dec66e98846e78cd730fe92c8a6b7: Deleting consensus metadata
I20260924 23:16:07.186367 2752 catalog_manager.cc:4841] TS 9b9dec66e98846e78cd730fe92c8a6b7 (127.1.4.195:34167): tablet 14160b534d1c4a48a5c96a09b40f9bc8 (table test-table [id=912b4363c0a14e8b89e424d548dee139]) successfully deleted
I20260924 23:16:07.187548 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 3120
I20260924 23:16:07.192516 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 2736
2026-09-24T23:16:07Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5720 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:16:07 dist-test-slave-s1hw krb5kdc[3428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:16:07 dist-test-slave-s1hw krb5kdc[3428](info): set up 2 sockets
Sep 24 23:16:07 dist-test-slave-s1hw krb5kdc[3428](info): commencing operation
krb5kdc: starting...
W20260924 23:16:08.151017 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:16:08 dist-test-slave-s1hw krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:16:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:16:08.266106 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:44673
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:33581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:44673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:08.345469 3441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:08.345685 3441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:08.345712 3441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:08.346865 3441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:08.346908 3441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:08.346927 3441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:08.346942 3441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:08.346957 3441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:08.348445 3441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:33581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:44673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:44673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:08.348698 3441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:08.348914 3441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:08.353657 3449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:08.353783 3447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:08.353746 3441 server_base.cc:1034] running on GCE node
W20260924 23:16:08.353757 3446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:08.354209 3441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:08.354483 3441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:08.355631 3441 hybrid_clock.cc:648] HybridClock initialized: now 1790291768355620 us; error 30 us; skew 500 ppm
Sep 24 23:16:08 dist-test-slave-s1hw krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291768, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:08.358081 3441 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:16:08.358528 3441 webserver.cc:460] Webserver started at http://127.1.4.254:35269/ using document root <none> and password file <none>
I20260924 23:16:08.358747 3441 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:08.358793 3441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:08.358916 3441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:08.359969 3441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "b9ee2ef1f74c4527937c14c4c5731728"
format_stamp: "Formatted at 2026-09-24 23:16:08 on dist-test-slave-s1hw"
server_key: "45b88b062ddf693218e016a7552c7271"
server_key_iv: "b31682169776273ca2e55d1d1f6f10fe"
server_key_version: "encryptionkey@0"
I20260924 23:16:08.360296 3441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "b9ee2ef1f74c4527937c14c4c5731728"
format_stamp: "Formatted at 2026-09-24 23:16:08 on dist-test-slave-s1hw"
server_key: "45b88b062ddf693218e016a7552c7271"
server_key_iv: "b31682169776273ca2e55d1d1f6f10fe"
server_key_version: "encryptionkey@0"
I20260924 23:16:08.361522 3441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:16:08.362293 3456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:08.362473 3441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:08.362553 3441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "b9ee2ef1f74c4527937c14c4c5731728"
format_stamp: "Formatted at 2026-09-24 23:16:08 on dist-test-slave-s1hw"
server_key: "45b88b062ddf693218e016a7552c7271"
server_key_iv: "b31682169776273ca2e55d1d1f6f10fe"
server_key_version: "encryptionkey@0"
I20260924 23:16:08.362613 3441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:08.386714 3441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:08.388895 3441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:08.389042 3441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:08.392812 3441 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:44673
I20260924 23:16:08.392846 3508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:44673 every 8 connection(s)
I20260924 23:16:08.393234 3441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:08.394178 3509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:08.396615 3509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728: Bootstrap starting.
I20260924 23:16:08.397222 3509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:08.397511 3509 log.cc:824] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:08.398218 3509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728: No bootstrap required, opened a new log
I20260924 23:16:08.399991 3509 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ee2ef1f74c4527937c14c4c5731728" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 44673 } }
I20260924 23:16:08.400146 3509 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:08.400169 3509 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ee2ef1f74c4527937c14c4c5731728, State: Initialized, Role: FOLLOWER
I20260924 23:16:08.400322 3509 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ee2ef1f74c4527937c14c4c5731728" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 44673 } }
I20260924 23:16:08.400394 3509 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:08.400430 3509 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:08.400480 3509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:08.400872 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 3441
I20260924 23:16:08.400965 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:08.401157 1043 external_mini_cluster.cc:1428] Setting key 6f92a12c07f5431832ca3c8d7f06585b
I20260924 23:16:08.401432 3509 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ee2ef1f74c4527937c14c4c5731728" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 44673 } }
I20260924 23:16:08.401551 3509 leader_election.cc:302] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9ee2ef1f74c4527937c14c4c5731728; no voters:
I20260924 23:16:08.401759 3509 leader_election.cc:288] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:08.401944 3513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:08.402069 3513 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [term 1 LEADER]: Becoming Leader. State: Replica: b9ee2ef1f74c4527937c14c4c5731728, State: Running, Role: LEADER
I20260924 23:16:08.402149 3509 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:08.402217 3513 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ee2ef1f74c4527937c14c4c5731728" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 44673 } }
I20260924 23:16:08.402652 3515 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9ee2ef1f74c4527937c14c4c5731728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ee2ef1f74c4527937c14c4c5731728" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 44673 } } }
I20260924 23:16:08.402797 3515 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:08.402982 3516 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9ee2ef1f74c4527937c14c4c5731728. Latest consensus state: current_term: 1 leader_uuid: "b9ee2ef1f74c4527937c14c4c5731728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ee2ef1f74c4527937c14c4c5731728" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 44673 } } }
I20260924 23:16:08.403358 3516 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728 [sys.catalog]: This master's current role is: LEADER
W20260924 23:16:08.403828 3530 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:16:08.403872 3530 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:16:08.403919 3523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:08.404111 3523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:08.405669 3523 catalog_manager.cc:1347] Generated new cluster ID: dac877da164b43c1a05071b1149dbf5b
I20260924 23:16:08.405726 3523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:16:08 dist-test-slave-s1hw krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:08.412603 3523 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:08.413184 3523 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:08.413976 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:08.401678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.1.4.254:44673)
0924 23:16:08.402609 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:08.402620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:08.402659 (+ 39us) server_negotiation.cc:407] Connection header received
0924 23:16:08.402786 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:08.402794 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:08.402857 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:08.403871 (+ 1014us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:08.403971 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:08.405324 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:08.406401 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:08.407007 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:08.407127 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:08.409958 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:08.409964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:08.409965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:08.409979 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:08.412134 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:08.412499 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:08.412504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:08.412504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:08.412624 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:08.412896 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:08.412897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:08.412897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:08.413211 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:08.413376 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:08.413618 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 23:16:08.413763 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":104,"threads_started":1}
I20260924 23:16:08.417621 3523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9ee2ef1f74c4527937c14c4c5731728: Generated new TSK 0
I20260924 23:16:08.417793 3523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:16:08.446892 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:44673
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:33581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:08.528173 3535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:08.528383 3535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:08.528406 3535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:08.529757 3535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:08.529810 3535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:08.529837 3535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:16:08.531401 3535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:33581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:44673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:08.531627 3535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:08.531838 3535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:08.536692 3541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:08.536680 3540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:08.536752 3535 server_base.cc:1034] running on GCE node
W20260924 23:16:08.536680 3543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:08.537122 3535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:08.537360 3535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:08.538502 3535 hybrid_clock.cc:648] HybridClock initialized: now 1790291768538480 us; error 36 us; skew 500 ppm
Sep 24 23:16:08 dist-test-slave-s1hw krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291768, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:08.541062 3535 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:16:08.541484 3535 webserver.cc:460] Webserver started at http://127.1.4.193:44781/ using document root <none> and password file <none>
I20260924 23:16:08.541661 3535 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:08.541697 3535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:08.541786 3535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:08.542778 3535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "a350f9f55ed6420ba190cdd7e94de3d4"
format_stamp: "Formatted at 2026-09-24 23:16:08 on dist-test-slave-s1hw"
server_key: "69b8194d6f4b90010df8621e0d26704e"
server_key_iv: "90f1a8e3c30d7fb2a44d71d7d9af52d8"
server_key_version: "encryptionkey@0"
I20260924 23:16:08.543087 3535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "a350f9f55ed6420ba190cdd7e94de3d4"
format_stamp: "Formatted at 2026-09-24 23:16:08 on dist-test-slave-s1hw"
server_key: "69b8194d6f4b90010df8621e0d26704e"
server_key_iv: "90f1a8e3c30d7fb2a44d71d7d9af52d8"
server_key_version: "encryptionkey@0"
I20260924 23:16:08.544302 3535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:08.545049 3550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:08.545202 3535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:08.545279 3535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "a350f9f55ed6420ba190cdd7e94de3d4"
format_stamp: "Formatted at 2026-09-24 23:16:08 on dist-test-slave-s1hw"
server_key: "69b8194d6f4b90010df8621e0d26704e"
server_key_iv: "90f1a8e3c30d7fb2a44d71d7d9af52d8"
server_key_version: "encryptionkey@0"
I20260924 23:16:08.545329 3535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:08.561422 3535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:08.563572 3535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:08.563687 3535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:08.563872 3535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:08.564145 3535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:08.564174 3535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:08.564194 3535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:08.564205 3535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:08.569777 3535 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:42427
I20260924 23:16:08.569831 3663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:42427 every 8 connection(s)
I20260924 23:16:08.570137 3535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:16:08.571578 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 3535
I20260924 23:16:08.571676 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:16:08.571839 1043 external_mini_cluster.cc:1428] Setting key 439233674561ba2b27d24834270c5a64
Sep 24 23:16:08 dist-test-slave-s1hw krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291768, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
I20260924 23:16:08.579797 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:08.570831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:41399 (local address 127.1.4.254:44673)
0924 23:16:08.570883 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:08.570885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:08.571065 (+ 180us) server_negotiation.cc:407] Connection header received
0924 23:16:08.571595 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:08.571596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:08.571605 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:08.571663 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:08.572826 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:08.573337 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:08.574424 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:08.574656 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:08.574713 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:08.577125 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:08.577127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:08.577127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:08.577134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:08.578396 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:08.578849 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:08.578849 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:08.578850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:08.578895 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:08.579153 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:08.579153 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:08.579153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:08.579255 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:08.579300 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:08.579657 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 23:16:08.579729 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:16:08.579984 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:08.570799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:44673 (local address 127.1.4.193:41399)
0924 23:16:08.570942 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:08.570984 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:08.571497 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:08.571740 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:08.571745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:08.572458 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:08.572736 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:08.572744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:08.573425 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:08.573426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:08.574334 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:08.574336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:08.574736 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:08.574737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:08.574885 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:08.575411 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:08.575421 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:08.577030 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:08.578476 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:08.578485 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:08.578487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:08.578780 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:08.578951 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:08.578952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:08.578953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:08.579090 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:08.579311 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:08.579318 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:08.579519 (+ 201us) client_negotiation.cc:769] Sending connection context
0924 23:16:08.579640 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 23:16:08.579777 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260924 23:16:08.580453 3664 heartbeater.cc:347] Connected to a master server at 127.1.4.254:44673
I20260924 23:16:08.580577 3664 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:08.580757 3664 heartbeater.cc:511] Master 127.1.4.254:44673 requested a full tablet report, sending...
I20260924 23:16:08.581236 3473 ts_manager.cc:194] Registered new tserver with Master: a350f9f55ed6420ba190cdd7e94de3d4 (127.1.4.193:42427)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
I20260924 23:16:08.581864 3473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:41399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:16:08.598320 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:44673
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:33581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:08.677772 3669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:08.677987 3669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:08.678017 3669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:08.679162 3669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:08.679205 3669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:08.679267 3669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:16:08.680771 3669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:33581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:44673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:08.681011 3669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:08.681223 3669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:08.686268 3677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:08.686316 3669 server_base.cc:1034] running on GCE node
W20260924 23:16:08.686264 3675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:08.686268 3674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:08.686668 3669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:08.686920 3669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:08.688068 3669 hybrid_clock.cc:648] HybridClock initialized: now 1790291768688047 us; error 30 us; skew 500 ppm
Sep 24 23:16:08 dist-test-slave-s1hw krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291768, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:08.690400 3669 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:16:08.690866 3669 webserver.cc:460] Webserver started at http://127.1.4.194:44717/ using document root <none> and password file <none>
I20260924 23:16:08.691109 3669 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:08.691160 3669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:08.691277 3669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:08.692337 3669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "eee5be0243184acc802c094014be8649"
format_stamp: "Formatted at 2026-09-24 23:16:08 on dist-test-slave-s1hw"
server_key: "a3bd4f60ff3a883169cb685014d6e5d2"
server_key_iv: "32959b6d0bca1ae691dcfb323a041070"
server_key_version: "encryptionkey@0"
I20260924 23:16:08.692664 3669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "eee5be0243184acc802c094014be8649"
format_stamp: "Formatted at 2026-09-24 23:16:08 on dist-test-slave-s1hw"
server_key: "a3bd4f60ff3a883169cb685014d6e5d2"
server_key_iv: "32959b6d0bca1ae691dcfb323a041070"
server_key_version: "encryptionkey@0"
I20260924 23:16:08.693898 3669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:16:08.694826 3684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:08.694993 3669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:16:08.695086 3669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "eee5be0243184acc802c094014be8649"
format_stamp: "Formatted at 2026-09-24 23:16:08 on dist-test-slave-s1hw"
server_key: "a3bd4f60ff3a883169cb685014d6e5d2"
server_key_iv: "32959b6d0bca1ae691dcfb323a041070"
server_key_version: "encryptionkey@0"
I20260924 23:16:08.695148 3669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:08.709836 3669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:08.711906 3669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:08.712044 3669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:08.712253 3669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:08.712559 3669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:08.712592 3669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:08.712620 3669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:08.712646 3669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:08.717952 3669 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:46049
I20260924 23:16:08.718026 3797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:46049 every 8 connection(s)
I20260924 23:16:08.718305 3669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:16:08.723057 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 3669
I20260924 23:16:08.723157 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:16:08.723325 1043 external_mini_cluster.cc:1428] Setting key 8997654ad510a21b43e1427a3efccff8
Sep 24 23:16:08 dist-test-slave-s1hw krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291768, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:08.727659 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:08.718971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:48947 (local address 127.1.4.254:44673)
0924 23:16:08.719071 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:08.719073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:08.719223 (+ 150us) server_negotiation.cc:407] Connection header received
0924 23:16:08.719728 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:08.719729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:08.719739 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:08.719778 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:08.720652 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:08.721139 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:08.722197 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:08.722387 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:08.722444 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:08.724767 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:08.724769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:08.724769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:08.724777 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:08.726191 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:08.726638 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:08.726640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:08.726640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:08.726699 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:08.727013 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:08.727015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:08.727015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:08.727132 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:08.727191 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:08.727526 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 23:16:08.727593 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 23:16:08.727952 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:08.718953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:44673 (local address 127.1.4.194:48947)
0924 23:16:08.719108 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:08.719148 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:08.719632 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:08.719836 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:08.719839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:08.720379 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:08.720583 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:08.720588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:08.721228 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:08.721229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:08.722117 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:08.722120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:08.722491 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:08.722492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:08.722605 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:08.723131 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:08.723140 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:08.724681 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:08.726263 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:08.726269 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:08.726271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:08.726550 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:08.726757 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:08.726758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:08.726758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:08.726885 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:08.727195 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:08.727200 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:08.727385 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 23:16:08.727509 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 23:16:08.727690 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260924 23:16:08.728497 3798 heartbeater.cc:347] Connected to a master server at 127.1.4.254:44673
I20260924 23:16:08.728586 3798 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:08.728760 3798 heartbeater.cc:511] Master 127.1.4.254:44673 requested a full tablet report, sending...
I20260924 23:16:08.729080 3473 ts_manager.cc:194] Registered new tserver with Master: eee5be0243184acc802c094014be8649 (127.1.4.194:46049)
I20260924 23:16:08.729437 3473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:48947
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:16:08.749768 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:44673
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:33581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:08.839653 3803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:08.839920 3803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:08.839953 3803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:08.841684 3803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:08.841740 3803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:08.841779 3803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:16:08.843256 3803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:33581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:44673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:08.843513 3803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:08.843835 3803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:08.849969 3808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:08.850047 3809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:08.850238 3803 server_base.cc:1034] running on GCE node
W20260924 23:16:08.850044 3811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:08.850605 3803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:08.850853 3803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:08.851989 3803 hybrid_clock.cc:648] HybridClock initialized: now 1790291768851974 us; error 36 us; skew 500 ppm
Sep 24 23:16:08 dist-test-slave-s1hw krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291768, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:08.854358 3803 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:16:08.854800 3803 webserver.cc:460] Webserver started at http://127.1.4.195:42009/ using document root <none> and password file <none>
I20260924 23:16:08.855015 3803 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:08.855064 3803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:08.855178 3803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:08.856352 3803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "2439cf68c7b84d1e981b2ce79ba1e3d1"
format_stamp: "Formatted at 2026-09-24 23:16:08 on dist-test-slave-s1hw"
server_key: "77466415c01b5c673d0e4535b6b93408"
server_key_iv: "befb753e1db00d6ba6a564aa150cf4cd"
server_key_version: "encryptionkey@0"
I20260924 23:16:08.856680 3803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "2439cf68c7b84d1e981b2ce79ba1e3d1"
format_stamp: "Formatted at 2026-09-24 23:16:08 on dist-test-slave-s1hw"
server_key: "77466415c01b5c673d0e4535b6b93408"
server_key_iv: "befb753e1db00d6ba6a564aa150cf4cd"
server_key_version: "encryptionkey@0"
I20260924 23:16:08.857870 3803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:16:08.858672 3818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:08.858867 3803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:08.858956 3803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "2439cf68c7b84d1e981b2ce79ba1e3d1"
format_stamp: "Formatted at 2026-09-24 23:16:08 on dist-test-slave-s1hw"
server_key: "77466415c01b5c673d0e4535b6b93408"
server_key_iv: "befb753e1db00d6ba6a564aa150cf4cd"
server_key_version: "encryptionkey@0"
I20260924 23:16:08.859014 3803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:08.874857 3803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:08.877058 3803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:08.877177 3803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:08.877379 3803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:08.877642 3803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:08.877672 3803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:08.877693 3803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:08.877704 3803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:08.883261 3803 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:45441
I20260924 23:16:08.883324 3931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:45441 every 8 connection(s)
I20260924 23:16:08.883625 3803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:16:08.884676 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 3803
I20260924 23:16:08.884776 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:16:08.884979 1043 external_mini_cluster.cc:1428] Setting key 5d6c4e3fea31764d17246f1f9c931e22
Sep 24 23:16:08 dist-test-slave-s1hw krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291768, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:08.892849 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:08.884335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:47353 (local address 127.1.4.254:44673)
0924 23:16:08.884418 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:08.884420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:08.884584 (+ 164us) server_negotiation.cc:407] Connection header received
0924 23:16:08.885234 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:08.885235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:08.885249 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:08.885310 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:08.886383 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:08.886884 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:08.887943 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:08.888149 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:08.888214 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:08.890233 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:08.890236 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:08.890236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:08.890242 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:08.891606 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:08.891976 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:08.891977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:08.891977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:08.892027 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:08.892265 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:08.892266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:08.892267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:08.892412 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:08.892474 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:08.892710 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 23:16:08.892781 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 23:16:08.892975 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:08.884285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:44673 (local address 127.1.4.195:47353)
0924 23:16:08.884434 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:08.884476 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:08.885123 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:08.885398 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:08.885403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:08.886077 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:08.886275 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:08.886280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:08.886965 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:08.886966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:08.887840 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:08.887842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:08.888219 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:08.888220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:08.888317 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:08.888791 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:08.888798 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:08.890139 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:08.891665 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:08.891670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:08.891673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:08.891918 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:08.892082 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:08.892082 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:08.892082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:08.892176 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:08.892468 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:08.892471 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:08.892600 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 23:16:08.892678 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 23:16:08.892797 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260924 23:16:08.893435 3932 heartbeater.cc:347] Connected to a master server at 127.1.4.254:44673
I20260924 23:16:08.893517 3932 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:08.893676 3932 heartbeater.cc:511] Master 127.1.4.254:44673 requested a full tablet report, sending...
I20260924 23:16:08.894047 3473 ts_manager.cc:194] Registered new tserver with Master: 2439cf68c7b84d1e981b2ce79ba1e3d1 (127.1.4.195:45441)
I20260924 23:16:08.894379 3473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:47353
I20260924 23:16:08.896571 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 23:16:08.906577 3514 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:08.905782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.1.4.254:44673)
0924 23:16:08.905854 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:08.905856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:08.905903 (+ 47us) server_negotiation.cc:407] Connection header received
0924 23:16:08.905969 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:08.905970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:08.905978 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:08.906017 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:08.906509 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56678: BlockingRecv error: recv got EOF from 127.0.0.1:56678 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 23:16:08.907488 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 3535
I20260924 23:16:08.911612 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 3669
I20260924 23:16:08.915320 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 3803
I20260924 23:16:08.919265 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 3441
2026-09-24T23:16:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1724 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:16:08 dist-test-slave-s1hw krb5kdc[3944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:16:08 dist-test-slave-s1hw krb5kdc[3944](info): set up 2 sockets
Sep 24 23:16:08 dist-test-slave-s1hw krb5kdc[3944](info): commencing operation
krb5kdc: starting...
W20260924 23:16:09.875308 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 23:16:10.101622 1043 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.226s user 0.001s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 23:16:10.317787 1043 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.216s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 23:16:10.474155 1043 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.156s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:16:10 dist-test-slave-s1hw krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:16:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:16:10.566777 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:36521
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:36521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:10.647282 3957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:10.647593 3957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:10.647629 3957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:10.648900 3957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:10.648948 3957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:10.648967 3957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:10.648983 3957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:10.648998 3957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:10.650599 3957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:36521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:36521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:10.650890 3957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:10.651116 3957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:10.656121 3962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:10.656132 3963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:10.656196 3957 server_base.cc:1034] running on GCE node
W20260924 23:16:10.656121 3965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:10.656543 3957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:10.656821 3957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:10.657981 3957 hybrid_clock.cc:648] HybridClock initialized: now 1790291770657964 us; error 31 us; skew 500 ppm
Sep 24 23:16:10 dist-test-slave-s1hw krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291770, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:10.663914 3957 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:16:10.664330 3957 webserver.cc:460] Webserver started at http://127.1.4.254:43789/ using document root <none> and password file <none>
I20260924 23:16:10.664516 3957 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:10.664556 3957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:10.664649 3957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:10.665695 3957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "3c5ba85b41c04c6bb0a71c954007c455"
format_stamp: "Formatted at 2026-09-24 23:16:10 on dist-test-slave-s1hw"
server_key: "28ab4ca694a6902c7f2c2e68ea7dfc53"
server_key_iv: "a6722d8b81ffe8633e925fd1133517b3"
server_key_version: "encryptionkey@0"
I20260924 23:16:10.665985 3957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "3c5ba85b41c04c6bb0a71c954007c455"
format_stamp: "Formatted at 2026-09-24 23:16:10 on dist-test-slave-s1hw"
server_key: "28ab4ca694a6902c7f2c2e68ea7dfc53"
server_key_iv: "a6722d8b81ffe8633e925fd1133517b3"
server_key_version: "encryptionkey@0"
I20260924 23:16:10.667225 3957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 23:16:10.667964 3972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:10.668097 3957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:10.668180 3957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "3c5ba85b41c04c6bb0a71c954007c455"
format_stamp: "Formatted at 2026-09-24 23:16:10 on dist-test-slave-s1hw"
server_key: "28ab4ca694a6902c7f2c2e68ea7dfc53"
server_key_iv: "a6722d8b81ffe8633e925fd1133517b3"
server_key_version: "encryptionkey@0"
I20260924 23:16:10.668238 3957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:10.684430 3957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:10.686568 3957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:10.686698 3957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:10.690287 3957 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:36521
I20260924 23:16:10.690323 4024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:36521 every 8 connection(s)
I20260924 23:16:10.690675 3957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:10.691493 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 3957
I20260924 23:16:10.691588 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:10.691601 4025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:10.691781 1043 external_mini_cluster.cc:1428] Setting key 0281668cbe8cba0655060442c057d679
I20260924 23:16:10.694752 4025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455: Bootstrap starting.
I20260924 23:16:10.695456 4025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:10.695717 4025 log.cc:824] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:10.696429 4025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455: No bootstrap required, opened a new log
I20260924 23:16:10.698266 4025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5ba85b41c04c6bb0a71c954007c455" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36521 } }
I20260924 23:16:10.698406 4025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:10.698438 4025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5ba85b41c04c6bb0a71c954007c455, State: Initialized, Role: FOLLOWER
Sep 24 23:16:10 dist-test-slave-s1hw krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:10.698575 4025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5ba85b41c04c6bb0a71c954007c455" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36521 } }
I20260924 23:16:10.698654 4025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:10.698689 4025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:10.698734 4025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:10.699337 4025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5ba85b41c04c6bb0a71c954007c455" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36521 } }
I20260924 23:16:10.699460 4025 leader_election.cc:302] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c5ba85b41c04c6bb0a71c954007c455; no voters:
I20260924 23:16:10.699625 4025 leader_election.cc:288] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:10.699663 4030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:10.699810 4030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5ba85b41c04c6bb0a71c954007c455, State: Running, Role: LEADER
I20260924 23:16:10.699875 4025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:10.699935 4030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5ba85b41c04c6bb0a71c954007c455" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36521 } }
I20260924 23:16:10.700380 4032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c5ba85b41c04c6bb0a71c954007c455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5ba85b41c04c6bb0a71c954007c455" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36521 } } }
I20260924 23:16:10.700450 4032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:10.700654 4031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c5ba85b41c04c6bb0a71c954007c455. Latest consensus state: current_term: 1 leader_uuid: "3c5ba85b41c04c6bb0a71c954007c455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5ba85b41c04c6bb0a71c954007c455" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36521 } } }
I20260924 23:16:10.700877 4031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:10.701059 4039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:10.701608 4039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:10.702472 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:10.692298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.1.4.254:36521)
0924 23:16:10.692466 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:10.692475 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:10.692540 (+ 65us) server_negotiation.cc:407] Connection header received
0924 23:16:10.693055 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:10.693064 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:10.693127 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:10.693273 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:10.693881 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:10.695208 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:10.696202 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:10.696773 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:10.696886 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:10.699068 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:10.699072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:10.699072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:10.699084 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:10.701091 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:10.701431 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:10.701434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:10.701435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:10.701525 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:10.701704 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:10.701705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:10.701705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:10.701928 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:10.702076 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:10.702167 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 23:16:10.702291 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260924 23:16:10.703146 4039 catalog_manager.cc:1347] Generated new cluster ID: 076b631e52114aa6b10f5a41b3e880a0
I20260924 23:16:10.703190 4039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:16:10.703305 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:16:10.711849 4039 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:10.712369 4039 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:10.718994 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:10.714892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.1.4.254:36521)
0924 23:16:10.714933 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:10.714934 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:10.715059 (+ 125us) server_negotiation.cc:407] Connection header received
0924 23:16:10.715132 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:10.715133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:10.715144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:10.715209 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:10.715635 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:10.716416 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:10.717440 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:10.717751 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:10.717818 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:10.718520 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:10.718521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:10.718521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:10.718754 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:10.718772 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:10.718779 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:10.718839 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:10.718883 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 23:16:10.718910 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:16:10.724344 4039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c5ba85b41c04c6bb0a71c954007c455: Generated new TSK 0
I20260924 23:16:10.724548 4039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:16:10.727622 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 3957
2026-09-24T23:16:10Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1807 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:16:10 dist-test-slave-s1hw krb5kdc[4058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:16:10 dist-test-slave-s1hw krb5kdc[4058](info): set up 2 sockets
Sep 24 23:16:10 dist-test-slave-s1hw krb5kdc[4058](info): commencing operation
krb5kdc: starting...
W20260924 23:16:11.674995 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:16:11 dist-test-slave-s1hw krb5kdc[4058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:16:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:16:11.786491 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:35047
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:35047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:11.867255 4071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:11.867516 4071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:11.867542 4071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:11.868681 4071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:11.868719 4071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:11.868733 4071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:11.868744 4071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:11.868755 4071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:11.870213 4071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:35047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:35047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:11.870430 4071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:11.870678 4071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:11.875345 4076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:11.875355 4077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:11.875521 4071 server_base.cc:1034] running on GCE node
W20260924 23:16:11.875357 4079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:11.875837 4071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:11.876112 4071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:11.877261 4071 hybrid_clock.cc:648] HybridClock initialized: now 1790291771877240 us; error 33 us; skew 500 ppm
Sep 24 23:16:11 dist-test-slave-s1hw krb5kdc[4058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291771, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:11.879698 4071 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:16:11.880159 4071 webserver.cc:460] Webserver started at http://127.1.4.254:46661/ using document root <none> and password file <none>
I20260924 23:16:11.880363 4071 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:11.880410 4071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:11.880546 4071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:11.881613 4071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "e3749c62ac2b4afda7bd4c0402f19a6c"
format_stamp: "Formatted at 2026-09-24 23:16:11 on dist-test-slave-s1hw"
server_key: "dc711b18127a181e1745f53ffbeb7ddb"
server_key_iv: "455a261c82a8f075326c32ca15a25421"
server_key_version: "encryptionkey@0"
I20260924 23:16:11.881943 4071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "e3749c62ac2b4afda7bd4c0402f19a6c"
format_stamp: "Formatted at 2026-09-24 23:16:11 on dist-test-slave-s1hw"
server_key: "dc711b18127a181e1745f53ffbeb7ddb"
server_key_iv: "455a261c82a8f075326c32ca15a25421"
server_key_version: "encryptionkey@0"
I20260924 23:16:11.883214 4071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:16:11.883965 4086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:11.884119 4071 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:11.884200 4071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "e3749c62ac2b4afda7bd4c0402f19a6c"
format_stamp: "Formatted at 2026-09-24 23:16:11 on dist-test-slave-s1hw"
server_key: "dc711b18127a181e1745f53ffbeb7ddb"
server_key_iv: "455a261c82a8f075326c32ca15a25421"
server_key_version: "encryptionkey@0"
I20260924 23:16:11.884267 4071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:11.906916 4071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:11.909133 4071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:11.909273 4071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:11.913038 4071 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:35047
I20260924 23:16:11.913086 4138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:35047 every 8 connection(s)
I20260924 23:16:11.913399 4071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:11.914309 4139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:11.917466 4139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c: Bootstrap starting.
I20260924 23:16:11.918216 4139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:11.918545 4139 log.cc:824] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:11.919389 4139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c: No bootstrap required, opened a new log
I20260924 23:16:11.921353 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 4071
I20260924 23:16:11.921444 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:11.921602 1043 external_mini_cluster.cc:1428] Setting key f65b3132385032343d6fdf15d1c157f1
I20260924 23:16:11.921505 4139 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3749c62ac2b4afda7bd4c0402f19a6c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35047 } }
I20260924 23:16:11.921682 4139 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:11.921722 4139 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3749c62ac2b4afda7bd4c0402f19a6c, State: Initialized, Role: FOLLOWER
I20260924 23:16:11.921868 4139 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3749c62ac2b4afda7bd4c0402f19a6c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35047 } }
I20260924 23:16:11.921949 4139 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:11.921979 4139 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:11.922017 4139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:11.922843 4139 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3749c62ac2b4afda7bd4c0402f19a6c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35047 } }
I20260924 23:16:11.922935 4139 leader_election.cc:302] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3749c62ac2b4afda7bd4c0402f19a6c; no voters:
I20260924 23:16:11.923079 4139 leader_election.cc:288] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:11.923116 4144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:11.923244 4144 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [term 1 LEADER]: Becoming Leader. State: Replica: e3749c62ac2b4afda7bd4c0402f19a6c, State: Running, Role: LEADER
I20260924 23:16:11.923311 4139 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:11.923337 4144 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3749c62ac2b4afda7bd4c0402f19a6c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35047 } }
I20260924 23:16:11.923710 4145 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3749c62ac2b4afda7bd4c0402f19a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3749c62ac2b4afda7bd4c0402f19a6c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35047 } } }
I20260924 23:16:11.923724 4146 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3749c62ac2b4afda7bd4c0402f19a6c. Latest consensus state: current_term: 1 leader_uuid: "e3749c62ac2b4afda7bd4c0402f19a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3749c62ac2b4afda7bd4c0402f19a6c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35047 } } }
I20260924 23:16:11.923820 4145 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:11.923826 4146 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:11.924033 4153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:11.924573 4153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:11.926036 4153 catalog_manager.cc:1347] Generated new cluster ID: 7805815d9add4bdfb5390851f01df99f
I20260924 23:16:11.926082 4153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:16:11 dist-test-slave-s1hw krb5kdc[4058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:11.942417 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:11.922085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47240 (local address 127.1.4.254:35047)
0924 23:16:11.922264 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:11.922275 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:11.931450 (+ 9175us) server_negotiation.cc:407] Connection header received
0924 23:16:11.931604 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:11.931612 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:11.931677 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:11.931818 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:11.932596 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:11.934023 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:11.935141 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:11.935836 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:11.935934 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:11.938702 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:11.938709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:11.938709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:11.938725 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:11.940692 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:11.941146 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:11.941149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:11.941150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:11.941275 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:11.941457 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:11.941458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:11.941458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:11.941796 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:11.941948 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:11.942025 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 23:16:11.942168 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260924 23:16:11.945583 4153 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:11.946108 4153 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:11.961923 4153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3749c62ac2b4afda7bd4c0402f19a6c: Generated new TSK 0
I20260924 23:16:11.962163 4153 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:16:11.999001 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:35047
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:12.079330 4165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:12.079583 4165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:12.079607 4165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:12.080740 4165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:12.080777 4165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:12.080802 4165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:16:12.082235 4165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:35047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:12.082424 4165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:12.082602 4165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:12.087329 4173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:12.087517 4165 server_base.cc:1034] running on GCE node
W20260924 23:16:12.087334 4171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:12.087334 4170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:12.087793 4165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:12.088052 4165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:12.089183 4165 hybrid_clock.cc:648] HybridClock initialized: now 1790291772089167 us; error 33 us; skew 500 ppm
Sep 24 23:16:12 dist-test-slave-s1hw krb5kdc[4058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291772, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:12.091629 4165 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:16:12.092044 4165 webserver.cc:460] Webserver started at http://127.1.4.193:41111/ using document root <none> and password file <none>
I20260924 23:16:12.092216 4165 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:12.092254 4165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:12.092343 4165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:12.093367 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "d22735fb68d443d59b0b25c5d6275526"
format_stamp: "Formatted at 2026-09-24 23:16:12 on dist-test-slave-s1hw"
server_key: "a4df4a680284675944e6e8bf6d33265a"
server_key_iv: "c6938942b39bc3bc2daa976d47c6bbd6"
server_key_version: "encryptionkey@0"
I20260924 23:16:12.093655 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "d22735fb68d443d59b0b25c5d6275526"
format_stamp: "Formatted at 2026-09-24 23:16:12 on dist-test-slave-s1hw"
server_key: "a4df4a680284675944e6e8bf6d33265a"
server_key_iv: "c6938942b39bc3bc2daa976d47c6bbd6"
server_key_version: "encryptionkey@0"
I20260924 23:16:12.094846 4165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:12.095561 4180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:12.095716 4165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:12.095799 4165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "d22735fb68d443d59b0b25c5d6275526"
format_stamp: "Formatted at 2026-09-24 23:16:12 on dist-test-slave-s1hw"
server_key: "a4df4a680284675944e6e8bf6d33265a"
server_key_iv: "c6938942b39bc3bc2daa976d47c6bbd6"
server_key_version: "encryptionkey@0"
I20260924 23:16:12.095862 4165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:12.115455 4165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:12.117504 4165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:12.117630 4165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:12.117835 4165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:12.118162 4165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:12.118196 4165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:12.118217 4165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:12.118238 4165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:12.123550 4165 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:39237
I20260924 23:16:12.123603 4293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:39237 every 8 connection(s)
I20260924 23:16:12.123920 4165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:16:12 dist-test-slave-s1hw krb5kdc[4058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291772, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:12.132355 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:12.124580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:55901 (local address 127.1.4.254:35047)
0924 23:16:12.124625 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:12.124628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:12.124847 (+ 219us) server_negotiation.cc:407] Connection header received
0924 23:16:12.125389 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:12.125391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:12.125403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:12.125458 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:12.126350 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:12.126864 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:12.127838 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:12.128038 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:12.128089 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:12.130035 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:12.130036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:12.130036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:12.130043 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:12.131223 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:12.131619 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:12.131620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:12.131620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:12.131673 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:12.131846 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:12.131847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:12.131847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:12.131936 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:12.131978 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:12.132211 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 23:16:12.132287 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:16:12.132494 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:12.124578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:35047 (local address 127.1.4.193:55901)
0924 23:16:12.124732 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:12.124778 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:12.125278 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:12.125522 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:12.125525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:12.126082 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:12.126276 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:12.126280 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:12.126930 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:12.126930 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:12.127775 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:12.127777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:12.128097 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:12.128097 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:12.128191 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:12.128656 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:12.128664 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:12.129952 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:12.131287 (+ 1335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:12.131292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:12.131294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:12.131548 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:12.131722 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:12.131722 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:12.131723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:12.131812 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:12.131980 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:12.131985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:12.132108 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 23:16:12.132188 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 23:16:12.132302 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260924 23:16:12.132953 4294 heartbeater.cc:347] Connected to a master server at 127.1.4.254:35047
I20260924 23:16:12.133062 4294 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:12.133201 4294 heartbeater.cc:511] Master 127.1.4.254:35047 requested a full tablet report, sending...
I20260924 23:16:12.133704 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 4165
I20260924 23:16:12.133702 4103 ts_manager.cc:194] Registered new tserver with Master: d22735fb68d443d59b0b25c5d6275526 (127.1.4.193:39237)
I20260924 23:16:12.133781 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:16:12.133950 1043 external_mini_cluster.cc:1428] Setting key 8ef5604228ae4d736eccc29547190c70
I20260924 23:16:12.134500 4103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:55901
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:16:12.157878 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:35047
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:12.236387 4299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:12.236594 4299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:12.236615 4299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:12.237731 4299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:12.237768 4299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:12.237792 4299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:16:12.239164 4299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:35047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:12.239354 4299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:12.239565 4299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:12.244549 4307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:12.244483 4304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:12.244563 4299 server_base.cc:1034] running on GCE node
W20260924 23:16:12.244526 4305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:12.244900 4299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:12.245133 4299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:12.246285 4299 hybrid_clock.cc:648] HybridClock initialized: now 1790291772246270 us; error 28 us; skew 500 ppm
Sep 24 23:16:12 dist-test-slave-s1hw krb5kdc[4058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291772, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:12.248726 4299 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:16:12.249208 4299 webserver.cc:460] Webserver started at http://127.1.4.194:33463/ using document root <none> and password file <none>
I20260924 23:16:12.249421 4299 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:12.249467 4299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:12.249585 4299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:12.250582 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "eeeed0c193cd43b09d0e131190284cfc"
format_stamp: "Formatted at 2026-09-24 23:16:12 on dist-test-slave-s1hw"
server_key: "3e41a226cf1b646071023d961ffbf562"
server_key_iv: "7cb85aaa4fe8927797df5f22474ea619"
server_key_version: "encryptionkey@0"
I20260924 23:16:12.250912 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "eeeed0c193cd43b09d0e131190284cfc"
format_stamp: "Formatted at 2026-09-24 23:16:12 on dist-test-slave-s1hw"
server_key: "3e41a226cf1b646071023d961ffbf562"
server_key_iv: "7cb85aaa4fe8927797df5f22474ea619"
server_key_version: "encryptionkey@0"
I20260924 23:16:12.252151 4299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:16:12.252771 4314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:12.252915 4299 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:12.253041 4299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "eeeed0c193cd43b09d0e131190284cfc"
format_stamp: "Formatted at 2026-09-24 23:16:12 on dist-test-slave-s1hw"
server_key: "3e41a226cf1b646071023d961ffbf562"
server_key_iv: "7cb85aaa4fe8927797df5f22474ea619"
server_key_version: "encryptionkey@0"
I20260924 23:16:12.253103 4299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:12.267340 4299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:12.269523 4299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:12.269659 4299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:12.269878 4299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:12.270190 4299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:12.270221 4299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:12.270251 4299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:12.270268 4299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:12.275759 4299 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:45175
I20260924 23:16:12.275799 4427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:45175 every 8 connection(s)
I20260924 23:16:12.276114 4299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:16:12 dist-test-slave-s1hw krb5kdc[4058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291772, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:12.282522 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 4299
I20260924 23:16:12.282608 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:16:12.282799 1043 external_mini_cluster.cc:1428] Setting key 146b880ce5314e4a5b2817bc35d1df48
I20260924 23:16:12.285539 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:12.276838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:44015 (local address 127.1.4.254:35047)
0924 23:16:12.276879 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:12.276881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:12.277174 (+ 293us) server_negotiation.cc:407] Connection header received
0924 23:16:12.277648 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:12.277650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:12.277660 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:12.277711 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:12.278592 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:12.279088 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:12.280325 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:12.280521 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:12.280577 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:12.282820 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:12.282821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:12.282822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:12.282830 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:12.284185 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:12.284638 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:12.284640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:12.284640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:12.284709 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:12.284936 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:12.284936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:12.284937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:12.285062 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:12.285133 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:12.285344 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 23:16:12.285432 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 23:16:12.285620 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:12.276841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:35047 (local address 127.1.4.194:44015)
0924 23:16:12.276991 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:12.277080 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:12.277546 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:12.277769 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:12.277772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:12.278318 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:12.278516 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:12.278522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:12.279197 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:12.279198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:12.280213 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:12.280216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:12.280601 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:12.280601 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:12.280736 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:12.281289 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:12.281298 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:12.282732 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:12.284256 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:12.284263 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:12.284265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:12.284555 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:12.284775 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:12.284776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:12.284776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:12.284883 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:12.285121 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:12.285124 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:12.285239 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 23:16:12.285313 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 23:16:12.285445 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260924 23:16:12.286068 4428 heartbeater.cc:347] Connected to a master server at 127.1.4.254:35047
I20260924 23:16:12.286163 4428 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:12.286334 4428 heartbeater.cc:511] Master 127.1.4.254:35047 requested a full tablet report, sending...
I20260924 23:16:12.286726 4103 ts_manager.cc:194] Registered new tserver with Master: eeeed0c193cd43b09d0e131190284cfc (127.1.4.194:45175)
I20260924 23:16:12.287127 4103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:44015
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:16:12.308038 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:35047
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:12.386636 4433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:12.386871 4433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:12.386904 4433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:12.388155 4433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:12.388200 4433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:12.388238 4433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:16:12.389650 4433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:35047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:12.389880 4433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:12.390085 4433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:16:12.395210 4433 server_base.cc:1034] running on GCE node
W20260924 23:16:12.395211 4441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:12.395208 4439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:12.395211 4438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:12.395718 4433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:12.396016 4433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:12.397179 4433 hybrid_clock.cc:648] HybridClock initialized: now 1790291772397168 us; error 34 us; skew 500 ppm
Sep 24 23:16:12 dist-test-slave-s1hw krb5kdc[4058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291772, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:12.399680 4433 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:16:12.400130 4433 webserver.cc:460] Webserver started at http://127.1.4.195:45791/ using document root <none> and password file <none>
I20260924 23:16:12.400336 4433 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:12.400382 4433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:12.400507 4433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:12.401561 4433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "da35924078264a6ca79c5a22da88e33d"
format_stamp: "Formatted at 2026-09-24 23:16:12 on dist-test-slave-s1hw"
server_key: "99ed8baef0c46e91e885e65f32f6ca9c"
server_key_iv: "647dc0f2d8318b3072a5a7cbb44fff8f"
server_key_version: "encryptionkey@0"
I20260924 23:16:12.401885 4433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "da35924078264a6ca79c5a22da88e33d"
format_stamp: "Formatted at 2026-09-24 23:16:12 on dist-test-slave-s1hw"
server_key: "99ed8baef0c46e91e885e65f32f6ca9c"
server_key_iv: "647dc0f2d8318b3072a5a7cbb44fff8f"
server_key_version: "encryptionkey@0"
I20260924 23:16:12.403178 4433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 23:16:12.403906 4448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:12.404065 4433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:12.404146 4433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "da35924078264a6ca79c5a22da88e33d"
format_stamp: "Formatted at 2026-09-24 23:16:12 on dist-test-slave-s1hw"
server_key: "99ed8baef0c46e91e885e65f32f6ca9c"
server_key_iv: "647dc0f2d8318b3072a5a7cbb44fff8f"
server_key_version: "encryptionkey@0"
I20260924 23:16:12.404213 4433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:12.412007 4433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:12.413738 4433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:12.413865 4433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:12.414067 4433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:12.414337 4433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:12.414368 4433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:12.414395 4433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:12.414418 4433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:12.419934 4433 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:42449
I20260924 23:16:12.419986 4561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:42449 every 8 connection(s)
I20260924 23:16:12.420280 4433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:16:12.422595 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 4433
I20260924 23:16:12.422696 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:16:12.422881 1043 external_mini_cluster.cc:1428] Setting key b3c7a184daee44bbc2afcc7518dce0b6
Sep 24 23:16:12 dist-test-slave-s1hw krb5kdc[4058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291772, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:12.429040 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:12.420956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:33457 (local address 127.1.4.254:35047)
0924 23:16:12.421011 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:12.421013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:12.421187 (+ 174us) server_negotiation.cc:407] Connection header received
0924 23:16:12.421677 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:12.421678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:12.421687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:12.421722 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:12.422740 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:12.423446 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:12.424631 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:12.424830 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:12.424882 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:12.426870 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:12.426871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:12.426871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:12.426877 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:12.427990 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:12.428326 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:12.428327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:12.428327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:12.428371 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:12.428557 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:12.428557 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:12.428558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:12.428653 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:12.428698 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:12.428914 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 23:16:12.428971 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 23:16:12.429204 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:12.420928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:35047 (local address 127.1.4.195:33457)
0924 23:16:12.421069 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:12.421109 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:12.421583 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:12.421783 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:12.421787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:12.422363 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:12.422647 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:12.422655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:12.423521 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:12.423522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:12.424564 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:12.424567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:12.424900 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:12.424901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:12.425088 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:12.425503 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:12.425510 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:12.426794 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:12.428057 (+ 1263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:12.428062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:12.428064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:12.428269 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:12.428413 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:12.428413 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:12.428413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:12.428502 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:12.428697 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:12.428700 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:12.428814 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 23:16:12.428892 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 23:16:12.429008 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260924 23:16:12.429621 4562 heartbeater.cc:347] Connected to a master server at 127.1.4.254:35047
I20260924 23:16:12.429706 4562 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:12.429847 4562 heartbeater.cc:511] Master 127.1.4.254:35047 requested a full tablet report, sending...
I20260924 23:16:12.430181 4103 ts_manager.cc:194] Registered new tserver with Master: da35924078264a6ca79c5a22da88e33d (127.1.4.195:42449)
I20260924 23:16:12.430531 4103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:33457
I20260924 23:16:12.434468 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 23:16:12 dist-test-slave-s1hw krb5kdc[4058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291772, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 23:16:12 dist-test-slave-s1hw krb5kdc[4058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291772, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:12.456708 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:12.449556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.1.4.254:35047)
0924 23:16:12.449643 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:12.449645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:12.449743 (+ 98us) server_negotiation.cc:407] Connection header received
0924 23:16:12.449833 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:12.449834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:12.449846 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:12.449910 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:12.450486 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:12.451086 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:12.451911 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:12.452104 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:12.452164 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:12.454388 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:12.454394 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:12.454394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:12.454402 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:12.455726 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:12.456066 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:12.456067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:12.456068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:12.456118 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:12.456251 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:12.456251 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:12.456252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:12.456427 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:12.456505 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:12.456583 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 23:16:12.456640 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 23:16:12.457022 4103 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47246
I20260924 23:16:12.457417 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 4165
I20260924 23:16:12.461750 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 4299
I20260924 23:16:12.465512 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 4433
I20260924 23:16:12.469517 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 4071
2026-09-24T23:16:12Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1742 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:16:12 dist-test-slave-s1hw krb5kdc[4574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:16:12 dist-test-slave-s1hw krb5kdc[4574](info): set up 2 sockets
Sep 24 23:16:12 dist-test-slave-s1hw krb5kdc[4574](info): commencing operation
krb5kdc: starting...
W20260924 23:16:13.414989 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:16:13 dist-test-slave-s1hw krb5kdc[4574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:16:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:16:13.533953 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:40935
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:40935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:13.613957 4587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:13.614171 4587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:13.614193 4587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:13.615358 4587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:13.615430 4587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:13.615443 4587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:13.615454 4587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:13.615468 4587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:13.616979 4587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:40935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:40935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:13.617213 4587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:13.617419 4587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:16:13.622522 4587 server_base.cc:1034] running on GCE node
W20260924 23:16:13.622534 4593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:13.622524 4595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:13.622534 4592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:13.622993 4587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:13.623283 4587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:13.624449 4587 hybrid_clock.cc:648] HybridClock initialized: now 1790291773624429 us; error 36 us; skew 500 ppm
Sep 24 23:16:13 dist-test-slave-s1hw krb5kdc[4574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291773, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:13.626850 4587 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:16:13.627326 4587 webserver.cc:460] Webserver started at http://127.1.4.254:46877/ using document root <none> and password file <none>
I20260924 23:16:13.627557 4587 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:13.627606 4587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:13.627736 4587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:13.628927 4587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "1c642fd0c35746ae8985628682fd4315"
format_stamp: "Formatted at 2026-09-24 23:16:13 on dist-test-slave-s1hw"
server_key: "788bfa5abc03534067c3170fb7721b6a"
server_key_iv: "e2adc2a64756a12a23d5397ed3fb6b0f"
server_key_version: "encryptionkey@0"
I20260924 23:16:13.629266 4587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "1c642fd0c35746ae8985628682fd4315"
format_stamp: "Formatted at 2026-09-24 23:16:13 on dist-test-slave-s1hw"
server_key: "788bfa5abc03534067c3170fb7721b6a"
server_key_iv: "e2adc2a64756a12a23d5397ed3fb6b0f"
server_key_version: "encryptionkey@0"
I20260924 23:16:13.630486 4587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:13.631155 4602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:13.631289 4587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:13.631395 4587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "1c642fd0c35746ae8985628682fd4315"
format_stamp: "Formatted at 2026-09-24 23:16:13 on dist-test-slave-s1hw"
server_key: "788bfa5abc03534067c3170fb7721b6a"
server_key_iv: "e2adc2a64756a12a23d5397ed3fb6b0f"
server_key_version: "encryptionkey@0"
I20260924 23:16:13.631464 4587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:13.647451 4587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:13.649252 4587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:13.649402 4587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:13.653442 4587 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:40935
I20260924 23:16:13.653475 4654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:40935 every 8 connection(s)
I20260924 23:16:13.653955 4587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:13.654580 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:13.656904 4655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315: Bootstrap starting.
I20260924 23:16:13.657459 4655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:13.657724 4655 log.cc:824] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:13.658390 4655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315: No bootstrap required, opened a new log
I20260924 23:16:13.658591 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 4587
I20260924 23:16:13.658690 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:13.658860 1043 external_mini_cluster.cc:1428] Setting key 52a1d0709629796a4de93d259d583140
I20260924 23:16:13.660782 4655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c642fd0c35746ae8985628682fd4315" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40935 } }
I20260924 23:16:13.660921 4655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:13.660953 4655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c642fd0c35746ae8985628682fd4315, State: Initialized, Role: FOLLOWER
I20260924 23:16:13.661091 4655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c642fd0c35746ae8985628682fd4315" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40935 } }
I20260924 23:16:13.661170 4655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:13.661206 4655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:13.661252 4655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:13.661904 4655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c642fd0c35746ae8985628682fd4315" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40935 } }
I20260924 23:16:13.661988 4655 leader_election.cc:302] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c642fd0c35746ae8985628682fd4315; no voters:
I20260924 23:16:13.662134 4655 leader_election.cc:288] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:13.662185 4660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:13.662320 4660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [term 1 LEADER]: Becoming Leader. State: Replica: 1c642fd0c35746ae8985628682fd4315, State: Running, Role: LEADER
I20260924 23:16:13.662364 4655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:13.662401 4660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c642fd0c35746ae8985628682fd4315" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40935 } }
I20260924 23:16:13.662714 4660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c642fd0c35746ae8985628682fd4315. Latest consensus state: current_term: 1 leader_uuid: "1c642fd0c35746ae8985628682fd4315" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c642fd0c35746ae8985628682fd4315" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40935 } } }
I20260924 23:16:13.662782 4660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:13.662724 4662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c642fd0c35746ae8985628682fd4315" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c642fd0c35746ae8985628682fd4315" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40935 } } }
I20260924 23:16:13.662961 4662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315 [sys.catalog]: This master's current role is: LEADER
W20260924 23:16:13.664047 4675 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:16:13.664090 4675 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:16:13.664176 4666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:13.664345 4666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:13.666136 4666 catalog_manager.cc:1347] Generated new cluster ID: 16e5135732c34564ad8d9a7b9f6b9867
I20260924 23:16:13.666188 4666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:16:13 dist-test-slave-s1hw krb5kdc[4574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:13.670979 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:13.659392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45688 (local address 127.1.4.254:40935)
0924 23:16:13.659562 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:13.659570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:13.659595 (+ 25us) server_negotiation.cc:407] Connection header received
0924 23:16:13.660757 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:13.660763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:13.660819 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:13.660942 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:13.661676 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:13.662951 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:13.663874 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:13.664477 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:13.664585 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:13.667152 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:13.667157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:13.667157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:13.667170 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:13.669218 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:13.669585 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:13.669588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:13.669588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:13.669715 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:13.669962 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:13.669963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:13.669964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:13.670327 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:13.670511 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:13.670583 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 23:16:13.670757 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260924 23:16:13.672765 4666 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:13.673493 4666 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:13.681666 4666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c642fd0c35746ae8985628682fd4315: Generated new TSK 0
I20260924 23:16:13.681862 4666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:16:13.714061 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40935
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:13.793906 4681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:13.794121 4681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:13.794144 4681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:13.795267 4681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:13.795300 4681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:13.795325 4681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:16:13.796829 4681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:13.797021 4681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:13.797190 4681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:13.801980 4686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:13.801975 4687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:13.802086 4689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:13.802323 4681 server_base.cc:1034] running on GCE node
I20260924 23:16:13.802502 4681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:13.802731 4681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:13.803866 4681 hybrid_clock.cc:648] HybridClock initialized: now 1790291773803854 us; error 29 us; skew 500 ppm
Sep 24 23:16:13 dist-test-slave-s1hw krb5kdc[4574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291773, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:13.806244 4681 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:16:13.806640 4681 webserver.cc:460] Webserver started at http://127.1.4.193:40063/ using document root <none> and password file <none>
I20260924 23:16:13.806807 4681 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:13.806844 4681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:13.806928 4681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:13.807932 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "3eda5214a53d4ce0b74577dd056dd2de"
format_stamp: "Formatted at 2026-09-24 23:16:13 on dist-test-slave-s1hw"
server_key: "752191f6ae43d693db6ddeeb5e6256a6"
server_key_iv: "dd5eee62f40daf4fd3efb63725f6c072"
server_key_version: "encryptionkey@0"
I20260924 23:16:13.808254 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "3eda5214a53d4ce0b74577dd056dd2de"
format_stamp: "Formatted at 2026-09-24 23:16:13 on dist-test-slave-s1hw"
server_key: "752191f6ae43d693db6ddeeb5e6256a6"
server_key_iv: "dd5eee62f40daf4fd3efb63725f6c072"
server_key_version: "encryptionkey@0"
I20260924 23:16:13.809418 4681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:16:13.810034 4696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:13.810174 4681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:13.810247 4681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "3eda5214a53d4ce0b74577dd056dd2de"
format_stamp: "Formatted at 2026-09-24 23:16:13 on dist-test-slave-s1hw"
server_key: "752191f6ae43d693db6ddeeb5e6256a6"
server_key_iv: "dd5eee62f40daf4fd3efb63725f6c072"
server_key_version: "encryptionkey@0"
I20260924 23:16:13.810310 4681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:13.818691 4681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:13.820752 4681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:13.820896 4681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:13.821164 4681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:13.821470 4681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:13.821503 4681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:13.821542 4681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:13.821564 4681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:13.826712 4681 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:39507
I20260924 23:16:13.826767 4809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:39507 every 8 connection(s)
I20260924 23:16:13.827050 4681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:16:13.828576 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 4681
I20260924 23:16:13.828687 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:16:13.828895 1043 external_mini_cluster.cc:1428] Setting key 5f0bbbdc8469fcb9f147f4c174487c8c
Sep 24 23:16:13 dist-test-slave-s1hw krb5kdc[4574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291773, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
I20260924 23:16:13.837651 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:13.827766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:54009 (local address 127.1.4.254:40935)
0924 23:16:13.827812 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:13.827814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:13.828118 (+ 304us) server_negotiation.cc:407] Connection header received
0924 23:16:13.829363 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:13.829364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:13.829381 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:13.829446 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:13.830278 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:13.831074 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:13.832284 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:13.832499 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:13.832559 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:13.834884 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:13.834885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:13.834886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:13.834897 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:13.836307 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:13.836688 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:13.836689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:13.836689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:13.836748 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:13.837012 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:13.837013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:13.837013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:13.837138 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:13.837195 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:13.837466 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 23:16:13.837556 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:16:13.837841 4812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:13.827725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40935 (local address 127.1.4.193:54009)
0924 23:16:13.827889 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:13.827924 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:13.829247 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:13.829525 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:13.829530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:13.829890 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:13.830174 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:13.830181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:13.831184 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:13.831185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:13.832202 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:13.832204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:13.832578 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:13.832580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:13.832710 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:13.833219 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:13.833227 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:13.834792 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:13.836385 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:13.836393 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:13.836395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:13.836622 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:13.836800 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:13.836800 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:13.836801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:13.836907 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:13.837195 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:13.837202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:13.837346 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 23:16:13.837436 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 23:16:13.837613 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260924 23:16:13.838455 4810 heartbeater.cc:347] Connected to a master server at 127.1.4.254:40935
I20260924 23:16:13.838562 4810 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:13.838754 4810 heartbeater.cc:511] Master 127.1.4.254:40935 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
I20260924 23:16:13.839470 4619 ts_manager.cc:194] Registered new tserver with Master: 3eda5214a53d4ce0b74577dd056dd2de (127.1.4.193:39507)
I20260924 23:16:13.840150 4619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:54009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:16:13.852398 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40935
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:13.931445 4815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:13.931669 4815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:13.931692 4815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:13.933022 4815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:13.933071 4815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:13.933099 4815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:16:13.934680 4815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:13.934928 4815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:13.935153 4815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:13.940189 4821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:13.940268 4823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:13.940198 4820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:13.940392 4815 server_base.cc:1034] running on GCE node
I20260924 23:16:13.940636 4815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:13.940862 4815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:13.942087 4815 hybrid_clock.cc:648] HybridClock initialized: now 1790291773942070 us; error 31 us; skew 500 ppm
Sep 24 23:16:13 dist-test-slave-s1hw krb5kdc[4574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291773, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:13.944435 4815 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:16:13.944839 4815 webserver.cc:460] Webserver started at http://127.1.4.194:42971/ using document root <none> and password file <none>
I20260924 23:16:13.945039 4815 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:13.945086 4815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:13.945207 4815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:13.946295 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "1b1d177f3f91427894f40f38a8308b58"
format_stamp: "Formatted at 2026-09-24 23:16:13 on dist-test-slave-s1hw"
server_key: "593fdf9695a2e8ebcee32e643867b477"
server_key_iv: "223fec3ee14f9a215f79e245d1317aa9"
server_key_version: "encryptionkey@0"
I20260924 23:16:13.946638 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "1b1d177f3f91427894f40f38a8308b58"
format_stamp: "Formatted at 2026-09-24 23:16:13 on dist-test-slave-s1hw"
server_key: "593fdf9695a2e8ebcee32e643867b477"
server_key_iv: "223fec3ee14f9a215f79e245d1317aa9"
server_key_version: "encryptionkey@0"
I20260924 23:16:13.947943 4815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:13.948577 4830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:13.948719 4815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:13.948796 4815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "1b1d177f3f91427894f40f38a8308b58"
format_stamp: "Formatted at 2026-09-24 23:16:13 on dist-test-slave-s1hw"
server_key: "593fdf9695a2e8ebcee32e643867b477"
server_key_iv: "223fec3ee14f9a215f79e245d1317aa9"
server_key_version: "encryptionkey@0"
I20260924 23:16:13.948851 4815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:13.968345 4815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:13.970382 4815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:13.970521 4815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:13.970755 4815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:13.971087 4815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:13.971127 4815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:13.971155 4815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:13.971172 4815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:13.976791 4815 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:41895
I20260924 23:16:13.976842 4943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:41895 every 8 connection(s)
I20260924 23:16:13.977149 4815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:16:13 dist-test-slave-s1hw krb5kdc[4574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291773, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:13.986023 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:13.977819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:51517 (local address 127.1.4.254:40935)
0924 23:16:13.977873 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:13.977876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:13.978156 (+ 280us) server_negotiation.cc:407] Connection header received
0924 23:16:13.979069 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:13.979071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:13.979083 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:13.979134 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:13.979801 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:13.980304 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:13.981271 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:13.981472 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:13.981528 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:13.983594 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:13.983595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:13.983596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:13.983604 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:13.984742 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:13.985160 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:13.985161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:13.985162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:13.985214 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:13.985429 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:13.985430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:13.985430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:13.985547 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:13.985602 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:13.985851 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 23:16:13.985932 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:16:13.986205 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:13.977798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40935 (local address 127.1.4.194:51517)
0924 23:16:13.978032 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:13.978069 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:13.978966 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:13.979203 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:13.979206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:13.979518 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:13.979710 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:13.979715 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:13.980390 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:13.980390 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:13.981188 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:13.981191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:13.981543 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:13.981544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:13.981657 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:13.982139 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:13.982148 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:13.983504 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:13.984817 (+ 1313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:13.984824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:13.984826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:13.985090 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:13.985267 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:13.985267 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:13.985268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:13.985365 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:13.985601 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:13.985607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:13.985740 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 23:16:13.985828 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 23:16:13.985994 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260924 23:16:13.986634 4944 heartbeater.cc:347] Connected to a master server at 127.1.4.254:40935
I20260924 23:16:13.986721 4944 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:13.986901 4944 heartbeater.cc:511] Master 127.1.4.254:40935 requested a full tablet report, sending...
I20260924 23:16:13.987033 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 4815
I20260924 23:16:13.987140 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:16:13.987259 4619 ts_manager.cc:194] Registered new tserver with Master: 1b1d177f3f91427894f40f38a8308b58 (127.1.4.194:41895)
I20260924 23:16:13.987308 1043 external_mini_cluster.cc:1428] Setting key 7315f5bcbf88c2c1e4c9044e124d9e5d
I20260924 23:16:13.987855 4619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:51517
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:16:14.010739 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40935
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:14.090909 4949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:14.091112 4949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:14.091135 4949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:14.092335 4949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:14.092382 4949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:14.092409 4949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:16:14.093977 4949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:14.094201 4949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:14.094422 4949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:14.099452 4955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:14.099488 4949 server_base.cc:1034] running on GCE node
W20260924 23:16:14.099452 4957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:14.099448 4954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:14.099907 4949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:14.100160 4949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:14.101337 4949 hybrid_clock.cc:648] HybridClock initialized: now 1790291774101318 us; error 35 us; skew 500 ppm
Sep 24 23:16:14 dist-test-slave-s1hw krb5kdc[4574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291774, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:14.103957 4949 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:16:14.104496 4949 webserver.cc:460] Webserver started at http://127.1.4.195:36657/ using document root <none> and password file <none>
I20260924 23:16:14.104709 4949 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:14.104763 4949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:14.104883 4949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:14.105899 4949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "a91e64cdc3ad482c9513969b4a7ac0ac"
format_stamp: "Formatted at 2026-09-24 23:16:14 on dist-test-slave-s1hw"
server_key: "e733ba12c16d4c1c9299349c89947abf"
server_key_iv: "25a39baf61464e562ed1dcc8165ff7d5"
server_key_version: "encryptionkey@0"
I20260924 23:16:14.106230 4949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "a91e64cdc3ad482c9513969b4a7ac0ac"
format_stamp: "Formatted at 2026-09-24 23:16:14 on dist-test-slave-s1hw"
server_key: "e733ba12c16d4c1c9299349c89947abf"
server_key_iv: "25a39baf61464e562ed1dcc8165ff7d5"
server_key_version: "encryptionkey@0"
I20260924 23:16:14.107512 4949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:14.108242 4964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:14.108400 4949 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:14.108481 4949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "a91e64cdc3ad482c9513969b4a7ac0ac"
format_stamp: "Formatted at 2026-09-24 23:16:14 on dist-test-slave-s1hw"
server_key: "e733ba12c16d4c1c9299349c89947abf"
server_key_iv: "25a39baf61464e562ed1dcc8165ff7d5"
server_key_version: "encryptionkey@0"
I20260924 23:16:14.108548 4949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:14.120765 4949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:14.122741 4949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:14.122882 4949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:14.123112 4949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:14.123426 4949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:14.123461 4949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:14.123492 4949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:14.123508 4949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:14.128963 4949 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:42621
I20260924 23:16:14.129016 5077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:42621 every 8 connection(s)
I20260924 23:16:14.129326 4949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
Sep 24 23:16:14 dist-test-slave-s1hw krb5kdc[4574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291774, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:14.135247 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 4949
I20260924 23:16:14.135346 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:16:14.135565 1043 external_mini_cluster.cc:1428] Setting key cd199038eb476636b8b31eb6a3be5095
I20260924 23:16:14.138378 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:14.129992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:47719 (local address 127.1.4.254:40935)
0924 23:16:14.130046 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:14.130048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:14.130240 (+ 192us) server_negotiation.cc:407] Connection header received
0924 23:16:14.131179 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:14.131181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:14.131193 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:14.131247 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:14.131927 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:14.132420 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:14.133398 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:14.133584 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:14.133640 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:14.135782 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:14.135784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:14.135784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:14.135792 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:14.137163 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:14.137576 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:14.137577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:14.137578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:14.137641 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:14.137858 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:14.137859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:14.137859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:14.137971 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:14.138051 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:14.138256 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 23:16:14.138316 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:16:14.138556 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:14.129961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40935 (local address 127.1.4.195:47719)
0924 23:16:14.130117 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:14.130159 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:14.131068 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:14.131335 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:14.131339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:14.131649 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:14.131837 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:14.131842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:14.132501 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:14.132502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:14.133317 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:14.133319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:14.133657 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:14.133658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:14.133803 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:14.134273 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:14.134282 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:14.135695 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:14.137229 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:14.137234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:14.137235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:14.137443 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:14.137699 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:14.137700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:14.137700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:14.137790 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:14.138044 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:14.138047 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:14.138160 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 23:16:14.138235 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 23:16:14.138349 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260924 23:16:14.138962 5078 heartbeater.cc:347] Connected to a master server at 127.1.4.254:40935
I20260924 23:16:14.139057 5078 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:14.139230 5078 heartbeater.cc:511] Master 127.1.4.254:40935 requested a full tablet report, sending...
I20260924 23:16:14.139559 4618 ts_manager.cc:194] Registered new tserver with Master: a91e64cdc3ad482c9513969b4a7ac0ac (127.1.4.195:42621)
I20260924 23:16:14.139926 4618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:47719
I20260924 23:16:14.147601 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 23:16:14 dist-test-slave-s1hw krb5kdc[4574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 23:16:14 dist-test-slave-s1hw krb5kdc[4574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.193@KRBTEST.COM
I20260924 23:16:14.169626 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:14.162076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.1.4.193:39507)
0924 23:16:14.162286 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:14.162292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:14.162305 (+ 13us) server_negotiation.cc:407] Connection header received
0924 23:16:14.162353 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:14.162355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:14.162371 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:14.162448 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:14.162982 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:14.163932 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:14.164661 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:14.165004 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:14.165062 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:14.167145 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:14.167155 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:14.167155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:14.167162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:14.168503 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:14.168805 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:14.168807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:14.168808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:14.168867 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:14.169027 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:14.169028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:14.169028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:14.169203 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:14.169348 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:14.169426 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 23:16:14.169502 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":98,"threads_started":1}
Sep 24 23:16:14 dist-test-slave-s1hw krb5kdc[4574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:14.176831 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:14.170777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.1.4.254:40935)
0924 23:16:14.170824 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:14.170826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:14.170897 (+ 71us) server_negotiation.cc:407] Connection header received
0924 23:16:14.170941 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:14.170942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:14.170953 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:14.171003 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:14.171449 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:14.171935 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:14.172595 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:14.172779 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:14.172831 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:14.174756 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:14.174757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:14.174758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:14.174764 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:14.176036 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:14.176294 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:14.176295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:14.176295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:14.176337 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:14.176453 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:14.176454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:14.176454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:14.176580 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:14.176645 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:14.176713 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 23:16:14.176767 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 23:16:14.177060 4618 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45704
I20260924 23:16:14.177368 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 4681
I20260924 23:16:14.181187 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 4815
I20260924 23:16:14.184700 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 4949
I20260924 23:16:14.188215 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 4587
2026-09-24T23:16:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1718 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:16:14 dist-test-slave-s1hw krb5kdc[5090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:16:14 dist-test-slave-s1hw krb5kdc[5090](info): set up 2 sockets
Sep 24 23:16:14 dist-test-slave-s1hw krb5kdc[5090](info): commencing operation
krb5kdc: starting...
W20260924 23:16:15.139174 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:16:15 dist-test-slave-s1hw krb5kdc[5090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:16:15.253706 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:36747
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:36747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:15.333014 5103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:15.333232 5103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:15.333254 5103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:15.334578 5103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:15.334635 5103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:15.334648 5103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:15.334661 5103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:15.334671 5103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:15.336428 5103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:36747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:36747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:15.336676 5103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:15.336894 5103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:15.341621 5108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:15.341645 5111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:15.341619 5109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:15.341675 5103 server_base.cc:1034] running on GCE node
I20260924 23:16:15.342085 5103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:15.342347 5103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:15.343457 5103 hybrid_clock.cc:648] HybridClock initialized: now 1790291775343443 us; error 30 us; skew 500 ppm
Sep 24 23:16:15 dist-test-slave-s1hw krb5kdc[5090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291775, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:15.345880 5103 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:16:15.346189 5103 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:15.346236 5103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:15.346349 5103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:15.347358 5103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "730734a33ba2484e9af312629552226e"
format_stamp: "Formatted at 2026-09-24 23:16:15 on dist-test-slave-s1hw"
server_key: "7fdfa6ad4e10617a444f2b82ff80a7ee"
server_key_iv: "29fdf510b0d953533a5dd1aa22a6b596"
server_key_version: "encryptionkey@0"
I20260924 23:16:15.347759 5103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "730734a33ba2484e9af312629552226e"
format_stamp: "Formatted at 2026-09-24 23:16:15 on dist-test-slave-s1hw"
server_key: "7fdfa6ad4e10617a444f2b82ff80a7ee"
server_key_iv: "29fdf510b0d953533a5dd1aa22a6b596"
server_key_version: "encryptionkey@0"
I20260924 23:16:15.348914 5103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:15.349546 5117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:15.349685 5103 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:15.349766 5103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "730734a33ba2484e9af312629552226e"
format_stamp: "Formatted at 2026-09-24 23:16:15 on dist-test-slave-s1hw"
server_key: "7fdfa6ad4e10617a444f2b82ff80a7ee"
server_key_iv: "29fdf510b0d953533a5dd1aa22a6b596"
server_key_version: "encryptionkey@0"
I20260924 23:16:15.349822 5103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:15.361644 5103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:15.363721 5103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:15.363852 5103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:15.367486 5103 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:36747
I20260924 23:16:15.367522 5169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:36747 every 8 connection(s)
I20260924 23:16:15.367779 5103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:15.368166 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 5103
I20260924 23:16:15.368249 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:15.368422 1043 external_mini_cluster.cc:1428] Setting key 55f58c87643a4b506e6501a8d5aa8dc4
I20260924 23:16:15.368736 5170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:15.371963 5170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e: Bootstrap starting.
I20260924 23:16:15.372634 5170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:15.372902 5170 log.cc:824] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:15.373591 5170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e: No bootstrap required, opened a new log
Sep 24 23:16:15 dist-test-slave-s1hw krb5kdc[5090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:15.375602 5170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730734a33ba2484e9af312629552226e" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36747 } }
I20260924 23:16:15.375768 5170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:15.375808 5170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730734a33ba2484e9af312629552226e, State: Initialized, Role: FOLLOWER
I20260924 23:16:15.375975 5170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730734a33ba2484e9af312629552226e" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36747 } }
I20260924 23:16:15.376067 5170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:15.376101 5170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:15.376147 5170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:15.376775 5170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730734a33ba2484e9af312629552226e" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36747 } }
I20260924 23:16:15.376894 5170 leader_election.cc:302] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 730734a33ba2484e9af312629552226e; no voters:
I20260924 23:16:15.377071 5170 leader_election.cc:288] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:15.377136 5175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:15.377311 5170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:15.377298 5175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [term 1 LEADER]: Becoming Leader. State: Replica: 730734a33ba2484e9af312629552226e, State: Running, Role: LEADER
I20260924 23:16:15.377439 5175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730734a33ba2484e9af312629552226e" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36747 } }
I20260924 23:16:15.377827 5176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "730734a33ba2484e9af312629552226e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730734a33ba2484e9af312629552226e" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36747 } } }
I20260924 23:16:15.377938 5176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:15.378093 5177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 730734a33ba2484e9af312629552226e. Latest consensus state: current_term: 1 leader_uuid: "730734a33ba2484e9af312629552226e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730734a33ba2484e9af312629552226e" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36747 } } }
I20260924 23:16:15.378150 5177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:15.378477 5184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:15.378963 5184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:15.379581 5173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.368896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.1.4.254:36747)
0924 23:16:15.369061 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:15.369071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:15.369096 (+ 25us) server_negotiation.cc:407] Connection header received
0924 23:16:15.369515 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:15.369521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:15.369566 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:15.369671 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:15.370428 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.371503 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.372547 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.373087 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.373183 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.375456 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:15.375461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:15.375461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:15.375474 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:15.377491 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:15.377923 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:15.377927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:15.378033 (+ 106us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:15.378137 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:15.378428 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:15.378428 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:15.378429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:15.378736 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:15.378879 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:15.379214 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 23:16:15.379345 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260924 23:16:15.380488 5184 catalog_manager.cc:1347] Generated new cluster ID: 8c416627b80443c9855f1866d2629821
I20260924 23:16:15.380543 5184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:16:15.393458 5184 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:15.394016 5184 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:15.404219 5184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e: Generated new TSK 0
I20260924 23:16:15.404424 5184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:16:15.434407 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:36747
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:15.515791 5196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:15.516023 5196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:15.516058 5196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:15.517258 5196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:15.517302 5196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:15.517339 5196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:16:15.518843 5196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:36747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:15.519188 5196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:15.519435 5196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:15.524621 5204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:15.524624 5201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:15.524684 5196 server_base.cc:1034] running on GCE node
W20260924 23:16:15.524614 5202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:15.525049 5196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:15.525322 5196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:15.526468 5196 hybrid_clock.cc:648] HybridClock initialized: now 1790291775526451 us; error 34 us; skew 500 ppm
Sep 24 23:16:15 dist-test-slave-s1hw krb5kdc[5090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291775, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:15.528949 5196 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:16:15.529425 5196 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:15.529487 5196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:15.529623 5196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:15.530776 5196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "5793ff0385e2401dae329af7feccf81f"
format_stamp: "Formatted at 2026-09-24 23:16:15 on dist-test-slave-s1hw"
server_key: "d2cb43180e2e10cb9f1c80b4f6170f7b"
server_key_iv: "4bc2382015f1cf19114e92033aa8e267"
server_key_version: "encryptionkey@0"
I20260924 23:16:15.531136 5196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "5793ff0385e2401dae329af7feccf81f"
format_stamp: "Formatted at 2026-09-24 23:16:15 on dist-test-slave-s1hw"
server_key: "d2cb43180e2e10cb9f1c80b4f6170f7b"
server_key_iv: "4bc2382015f1cf19114e92033aa8e267"
server_key_version: "encryptionkey@0"
I20260924 23:16:15.532542 5196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:16:15.533308 5210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:15.533479 5196 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:16:15.533565 5196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "5793ff0385e2401dae329af7feccf81f"
format_stamp: "Formatted at 2026-09-24 23:16:15 on dist-test-slave-s1hw"
server_key: "d2cb43180e2e10cb9f1c80b4f6170f7b"
server_key_iv: "4bc2382015f1cf19114e92033aa8e267"
server_key_version: "encryptionkey@0"
I20260924 23:16:15.533640 5196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:15.553736 5196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:15.555891 5196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:15.556047 5196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:15.556293 5196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:15.556615 5196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:15.556651 5196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:15.556685 5196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:15.556708 5196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:15.562726 5196 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:36727
I20260924 23:16:15.562762 5323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:36727 every 8 connection(s)
I20260924 23:16:15.563028 5196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:16:15 dist-test-slave-s1hw krb5kdc[5090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291775, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:15.569710 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 5196
I20260924 23:16:15.569820 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:16:15.569998 1043 external_mini_cluster.cc:1428] Setting key f8e1693224043ae1b536aa9edc3d2551
I20260924 23:16:15.572148 5173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.563707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:33681 (local address 127.1.4.254:36747)
0924 23:16:15.563794 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:15.563797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:15.563973 (+ 176us) server_negotiation.cc:407] Connection header received
0924 23:16:15.564446 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:15.564446 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:15.564457 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:15.564503 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:15.565441 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.565932 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.566959 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.567170 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.567222 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.569299 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:15.569300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:15.569300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:15.569307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:15.570749 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:15.571110 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:15.571111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:15.571112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:15.571172 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:15.571384 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:15.571385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:15.571385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:15.571510 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:15.571569 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:15.571952 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 23:16:15.572069 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 23:16:15.572175 5326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.563687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:36747 (local address 127.1.4.193:33681)
0924 23:16:15.563839 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:15.563887 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:15.564366 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:15.564571 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:15.564574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:15.564828 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:15.565341 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:15.565346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.566035 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.566035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:15.566874 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:15.566877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.567239 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.567240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:15.567343 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.567818 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:15.567827 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:15.569210 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:15.570820 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:15.570826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:15.570828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:15.571044 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:15.571232 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:15.571233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:15.571233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:15.571320 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:15.571560 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:15.571564 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:15.571687 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 23:16:15.571780 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 23:16:15.571945 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260924 23:16:15.572780 5324 heartbeater.cc:347] Connected to a master server at 127.1.4.254:36747
I20260924 23:16:15.572865 5324 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:15.573046 5324 heartbeater.cc:511] Master 127.1.4.254:36747 requested a full tablet report, sending...
I20260924 23:16:15.573562 5134 ts_manager.cc:194] Registered new tserver with Master: 5793ff0385e2401dae329af7feccf81f (127.1.4.193:36727)
I20260924 23:16:15.574394 5134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:33681
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:16:15.593789 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:36747
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:15.676121 5329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:15.676330 5329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:15.676353 5329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:15.677529 5329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:15.677568 5329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:15.677594 5329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:16:15.679080 5329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:36747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:15.679306 5329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:15.679527 5329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:15.684178 5335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:15.684242 5329 server_base.cc:1034] running on GCE node
W20260924 23:16:15.684199 5334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:15.684178 5337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:15.684607 5329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:15.684832 5329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:15.686059 5329 hybrid_clock.cc:648] HybridClock initialized: now 1790291775686039 us; error 30 us; skew 500 ppm
Sep 24 23:16:15 dist-test-slave-s1hw krb5kdc[5090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291775, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:15.691753 5329 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:16:15.692073 5329 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:15.692121 5329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:15.692241 5329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:15.693261 5329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "ffbcfafb94d545e39565faf2ab3f9614"
format_stamp: "Formatted at 2026-09-24 23:16:15 on dist-test-slave-s1hw"
server_key: "baaceaa1ec30f375298d56c0cd285e3a"
server_key_iv: "79d8809c4c93e7d38aafa800871b92c9"
server_key_version: "encryptionkey@0"
I20260924 23:16:15.693581 5329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "ffbcfafb94d545e39565faf2ab3f9614"
format_stamp: "Formatted at 2026-09-24 23:16:15 on dist-test-slave-s1hw"
server_key: "baaceaa1ec30f375298d56c0cd285e3a"
server_key_iv: "79d8809c4c93e7d38aafa800871b92c9"
server_key_version: "encryptionkey@0"
I20260924 23:16:15.694747 5329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 23:16:15.695406 5343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:15.695588 5329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:15.695678 5329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "ffbcfafb94d545e39565faf2ab3f9614"
format_stamp: "Formatted at 2026-09-24 23:16:15 on dist-test-slave-s1hw"
server_key: "baaceaa1ec30f375298d56c0cd285e3a"
server_key_iv: "79d8809c4c93e7d38aafa800871b92c9"
server_key_version: "encryptionkey@0"
I20260924 23:16:15.695741 5329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:15.710909 5329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:15.713006 5329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:15.713140 5329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:15.713348 5329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:15.713644 5329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:15.713678 5329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:15.713706 5329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:15.713730 5329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:15.718917 5329 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:39429
I20260924 23:16:15.718958 5456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:39429 every 8 connection(s)
I20260924 23:16:15.719199 5329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:16:15 dist-test-slave-s1hw krb5kdc[5090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291775, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:15.727934 5173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.719849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:34133 (local address 127.1.4.254:36747)
0924 23:16:15.719912 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:15.719914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:15.720132 (+ 218us) server_negotiation.cc:407] Connection header received
0924 23:16:15.720574 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:15.720575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:15.720585 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:15.720644 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:15.721587 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.722083 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.723045 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.723237 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.723301 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.725402 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:15.725404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:15.725404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:15.725444 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:15.726666 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:15.727061 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:15.727062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:15.727062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:15.727127 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:15.727336 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:15.727337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:15.727337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:15.727502 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:15.727556 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:15.727784 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 23:16:15.727860 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 23:16:15.728078 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.719818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:36747 (local address 127.1.4.194:34133)
0924 23:16:15.719973 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:15.720024 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:15.720487 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:15.720705 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:15.720709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:15.720982 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:15.721490 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:15.721495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.722150 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.722151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:15.722947 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:15.722950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.723322 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.723323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:15.723490 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.723949 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:15.723957 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:15.725309 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:15.726739 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:15.726745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:15.726747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:15.726948 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:15.727182 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:15.727183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:15.727183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:15.727284 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:15.727552 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:15.727556 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:15.727671 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 23:16:15.727752 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 23:16:15.727871 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260924 23:16:15.728314 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 5329
I20260924 23:16:15.728405 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:16:15.728590 1043 external_mini_cluster.cc:1428] Setting key 9086c08bc61ad95f03a77ceae7027410
I20260924 23:16:15.728572 5457 heartbeater.cc:347] Connected to a master server at 127.1.4.254:36747
I20260924 23:16:15.728657 5457 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:15.728818 5457 heartbeater.cc:511] Master 127.1.4.254:36747 requested a full tablet report, sending...
I20260924 23:16:15.729313 5134 ts_manager.cc:194] Registered new tserver with Master: ffbcfafb94d545e39565faf2ab3f9614 (127.1.4.194:39429)
I20260924 23:16:15.729688 5134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:34133
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:16:15.751549 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:36747
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:15.831045 5462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:15.831264 5462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:15.831290 5462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:15.832461 5462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:15.832500 5462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:15.832525 5462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:16:15.834010 5462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:36747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:15.834210 5462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:15.834394 5462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:15.839294 5468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:15.839327 5467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:15.839381 5462 server_base.cc:1034] running on GCE node
W20260924 23:16:15.839311 5470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:15.839736 5462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:15.839984 5462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:15.841132 5462 hybrid_clock.cc:648] HybridClock initialized: now 1790291775841118 us; error 28 us; skew 500 ppm
Sep 24 23:16:15 dist-test-slave-s1hw krb5kdc[5090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291775, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:15.843670 5462 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:16:15.844030 5462 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:15.844126 5462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:15.844255 5462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:15.845296 5462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "091fb4c9d1e64c7d9f5571282dde7fdd"
format_stamp: "Formatted at 2026-09-24 23:16:15 on dist-test-slave-s1hw"
server_key: "5686f657e0c9de77f0a1a47b5f21d116"
server_key_iv: "42c7e6de552f6806b7226e83dcc3cc44"
server_key_version: "encryptionkey@0"
I20260924 23:16:15.845624 5462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "091fb4c9d1e64c7d9f5571282dde7fdd"
format_stamp: "Formatted at 2026-09-24 23:16:15 on dist-test-slave-s1hw"
server_key: "5686f657e0c9de77f0a1a47b5f21d116"
server_key_iv: "42c7e6de552f6806b7226e83dcc3cc44"
server_key_version: "encryptionkey@0"
I20260924 23:16:15.846820 5462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:15.847549 5476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:15.847711 5462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:15.847791 5462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "091fb4c9d1e64c7d9f5571282dde7fdd"
format_stamp: "Formatted at 2026-09-24 23:16:15 on dist-test-slave-s1hw"
server_key: "5686f657e0c9de77f0a1a47b5f21d116"
server_key_iv: "42c7e6de552f6806b7226e83dcc3cc44"
server_key_version: "encryptionkey@0"
I20260924 23:16:15.847857 5462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:15.872319 5462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:15.874586 5462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:15.874754 5462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:15.875025 5462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:15.875396 5462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:15.875442 5462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:15.875476 5462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:15.875504 5462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:15.881767 5462 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:39819
I20260924 23:16:15.881803 5589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:39819 every 8 connection(s)
I20260924 23:16:15.882083 5462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:16:15.886279 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 5462
I20260924 23:16:15.886380 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:16:15.886590 1043 external_mini_cluster.cc:1428] Setting key 7cacdc7dcae3f45dda8b8e51750bfb3c
Sep 24 23:16:15 dist-test-slave-s1hw krb5kdc[5090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291775, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:15.891479 5173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.882777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:52875 (local address 127.1.4.254:36747)
0924 23:16:15.882841 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:15.882843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:15.883039 (+ 196us) server_negotiation.cc:407] Connection header received
0924 23:16:15.883626 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:15.883627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:15.883641 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:15.883688 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:15.884622 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.885118 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.886100 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.886402 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.886465 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.888801 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:15.888803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:15.888803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:15.888810 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:15.890086 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:15.890459 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:15.890461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:15.890461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:15.890523 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:15.890801 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:15.890801 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:15.890802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:15.890990 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:15.891050 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:15.891290 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 23:16:15.891383 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 23:16:15.891683 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.882747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:36747 (local address 127.1.4.195:52875)
0924 23:16:15.882908 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:15.882944 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:15.883529 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:15.883769 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:15.883774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:15.884039 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:15.884532 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:15.884537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.885198 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.885199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:15.886017 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:15.886019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.886501 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.886502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:15.886620 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.887160 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:15.887169 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:15.888699 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:15.890160 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:15.890165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:15.890167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:15.890369 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:15.890630 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:15.890631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:15.890631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:15.890743 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:15.891061 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:15.891065 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:15.891188 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 23:16:15.891272 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 23:16:15.891428 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260924 23:16:15.892145 5590 heartbeater.cc:347] Connected to a master server at 127.1.4.254:36747
I20260924 23:16:15.892230 5590 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:15.892374 5590 heartbeater.cc:511] Master 127.1.4.254:36747 requested a full tablet report, sending...
I20260924 23:16:15.892717 5134 ts_manager.cc:194] Registered new tserver with Master: 091fb4c9d1e64c7d9f5571282dde7fdd (127.1.4.195:39819)
I20260924 23:16:15.893127 5134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:52875
I20260924 23:16:15.898121 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:16:15.904815 5173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.899350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.1.4.254:36747)
0924 23:16:15.899425 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:15.899428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:15.899525 (+ 97us) server_negotiation.cc:407] Connection header received
0924 23:16:15.899566 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:15.899567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:15.899575 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:15.899618 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:15.900091 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.900638 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.901321 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.901490 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.901546 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.902558 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:15.902559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:15.902560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:15.902566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:15.903776 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:15.904117 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:15.904118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:15.904118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:15.904167 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:15.904359 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:15.904359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:15.904360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:15.904496 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:15.904587 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:15.904686 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 23:16:15.904750 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 23:16:15.906239 5134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:16:15.907814 5134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:15.914054 5602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.909958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:39429 (local address 127.0.0.1:53744)
0924 23:16:15.910115 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:15.910145 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:15.910207 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:15.910699 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:15.910700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:15.910702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:15.910750 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:15.910752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.911734 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.911735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:15.912937 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:15.912940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.913749 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.913750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:15.913830 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.913846 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 23:16:15.913927 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 23:16:15.913978 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":45,"threads_started":1}
I20260924 23:16:15.914080 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.910344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.1.4.194:39429)
0924 23:16:15.910475 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:15.910480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:15.910490 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:16:15.910530 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:15.910532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:15.910547 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:15.910823 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:15.910908 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.911649 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.913014 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.913668 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.913740 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.913800 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:15.913944 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 23:16:15.913988 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260924 23:16:15.914165 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.909971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:36727 (local address 127.0.0.1:44030)
0924 23:16:15.910296 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:15.910306 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:15.910354 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:15.910662 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:15.910665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:15.910667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:15.910739 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:15.910741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.912118 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.912119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:15.913137 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:15.913139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.913975 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.913976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:15.914036 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.914039 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:15.914066 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 23:16:15.914106 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":92,"threads_started":1}
I20260924 23:16:15.914443 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.910089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.1.4.193:36727)
0924 23:16:15.910464 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:15.910468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:15.910477 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:15.910518 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:15.910519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:15.910532 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:15.910811 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:15.910885 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.912025 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.913206 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.913888 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.914258 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.914327 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:15.914350 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 23:16:15.914382 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":75,"threads_started":1}
I20260924 23:16:15.914885 5391 tablet_service.cc:1467] Processing CreateTablet for tablet b0f9d94ea6164d81aa5eaaafef7df359 (DEFAULT_TABLE table=test-table [id=f8517fb569ba4ad1abc995b8c3d9be23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:15.915040 5258 tablet_service.cc:1467] Processing CreateTablet for tablet b0f9d94ea6164d81aa5eaaafef7df359 (DEFAULT_TABLE table=test-table [id=f8517fb569ba4ad1abc995b8c3d9be23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:15.915244 5391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f9d94ea6164d81aa5eaaafef7df359. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:15.915302 5258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f9d94ea6164d81aa5eaaafef7df359. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:15.915546 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.910168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:39819 (local address 127.0.0.1:37204)
0924 23:16:15.910496 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:15.910506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:15.910557 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:15.910782 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:15.910782 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:15.910784 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:15.910821 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:15.910823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.913203 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.913203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:15.913946 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:15.913947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.915293 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.915294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:15.915384 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.915388 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:15.915445 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 23:16:15.915485 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":91,"threads_started":1}
I20260924 23:16:15.915731 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.910269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37204 (local address 127.1.4.195:39819)
0924 23:16:15.910423 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:15.910428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:15.910563 (+ 135us) server_negotiation.cc:407] Connection header received
0924 23:16:15.910624 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:15.910626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:15.910640 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:15.910704 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:15.910890 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.912013 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.914025 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.915142 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.915205 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.915259 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:15.915474 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 23:16:15.915523 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260924 23:16:15.918452 5608 tablet_bootstrap.cc:492] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614: Bootstrap starting.
I20260924 23:16:15.918618 5607 tablet_bootstrap.cc:492] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f: Bootstrap starting.
I20260924 23:16:15.919229 5608 tablet_bootstrap.cc:654] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:15.919229 5607 tablet_bootstrap.cc:654] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:15.919186 5523 tablet_service.cc:1467] Processing CreateTablet for tablet b0f9d94ea6164d81aa5eaaafef7df359 (DEFAULT_TABLE table=test-table [id=f8517fb569ba4ad1abc995b8c3d9be23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:15.919471 5523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f9d94ea6164d81aa5eaaafef7df359. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:15.919885 5607 log.cc:824] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:15.919878 5608 log.cc:824] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:15.920599 5608 tablet_bootstrap.cc:492] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614: No bootstrap required, opened a new log
I20260924 23:16:15.920599 5607 tablet_bootstrap.cc:492] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f: No bootstrap required, opened a new log
I20260924 23:16:15.920666 5608 ts_tablet_manager.cc:1397] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:16:15.920666 5607 ts_tablet_manager.cc:1397] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 23:16:15.922209 5611 tablet_bootstrap.cc:492] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd: Bootstrap starting.
I20260924 23:16:15.922561 5607 raft_consensus.cc:348] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffbcfafb94d545e39565faf2ab3f9614" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 39429 } } peers { permanent_uuid: "5793ff0385e2401dae329af7feccf81f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36727 } } peers { permanent_uuid: "091fb4c9d1e64c7d9f5571282dde7fdd" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39819 } }
I20260924 23:16:15.922561 5608 raft_consensus.cc:348] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffbcfafb94d545e39565faf2ab3f9614" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 39429 } } peers { permanent_uuid: "5793ff0385e2401dae329af7feccf81f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36727 } } peers { permanent_uuid: "091fb4c9d1e64c7d9f5571282dde7fdd" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39819 } }
I20260924 23:16:15.922710 5611 tablet_bootstrap.cc:654] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:15.922731 5607 raft_consensus.cc:374] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:15.922731 5608 raft_consensus.cc:374] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:15.922766 5608 raft_consensus.cc:729] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffbcfafb94d545e39565faf2ab3f9614, State: Initialized, Role: FOLLOWER
I20260924 23:16:15.922766 5607 raft_consensus.cc:729] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5793ff0385e2401dae329af7feccf81f, State: Initialized, Role: FOLLOWER
I20260924 23:16:15.922942 5608 consensus_queue.cc:260] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffbcfafb94d545e39565faf2ab3f9614" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 39429 } } peers { permanent_uuid: "5793ff0385e2401dae329af7feccf81f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36727 } } peers { permanent_uuid: "091fb4c9d1e64c7d9f5571282dde7fdd" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39819 } }
I20260924 23:16:15.922979 5607 consensus_queue.cc:260] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffbcfafb94d545e39565faf2ab3f9614" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 39429 } } peers { permanent_uuid: "5793ff0385e2401dae329af7feccf81f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36727 } } peers { permanent_uuid: "091fb4c9d1e64c7d9f5571282dde7fdd" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39819 } }
I20260924 23:16:15.923157 5612 raft_consensus.cc:482] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:15.923242 5608 ts_tablet_manager.cc:1428] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 23:16:15.923242 5607 ts_tablet_manager.cc:1428] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 23:16:15.923255 5611 log.cc:824] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:15.923382 5457 heartbeater.cc:503] Master 127.1.4.254:36747 was elected leader, sending a full tablet report...
I20260924 23:16:15.923382 5324 heartbeater.cc:503] Master 127.1.4.254:36747 was elected leader, sending a full tablet report...
I20260924 23:16:15.923264 5612 raft_consensus.cc:504] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffbcfafb94d545e39565faf2ab3f9614" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 39429 } } peers { permanent_uuid: "5793ff0385e2401dae329af7feccf81f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36727 } } peers { permanent_uuid: "091fb4c9d1e64c7d9f5571282dde7fdd" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39819 } }
I20260924 23:16:15.923741 5612 leader_election.cc:288] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffbcfafb94d545e39565faf2ab3f9614 (127.1.4.194:39429), 091fb4c9d1e64c7d9f5571282dde7fdd (127.1.4.195:39819)
I20260924 23:16:15.924388 5611 tablet_bootstrap.cc:492] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd: No bootstrap required, opened a new log
I20260924 23:16:15.924465 5611 ts_tablet_manager.cc:1397] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:16:15.926043 5611 raft_consensus.cc:348] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffbcfafb94d545e39565faf2ab3f9614" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 39429 } } peers { permanent_uuid: "5793ff0385e2401dae329af7feccf81f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36727 } } peers { permanent_uuid: "091fb4c9d1e64c7d9f5571282dde7fdd" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39819 } }
I20260924 23:16:15.926184 5611 raft_consensus.cc:374] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:15.926213 5611 raft_consensus.cc:729] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091fb4c9d1e64c7d9f5571282dde7fdd, State: Initialized, Role: FOLLOWER
I20260924 23:16:15.926327 5611 consensus_queue.cc:260] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffbcfafb94d545e39565faf2ab3f9614" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 39429 } } peers { permanent_uuid: "5793ff0385e2401dae329af7feccf81f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36727 } } peers { permanent_uuid: "091fb4c9d1e64c7d9f5571282dde7fdd" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39819 } }
I20260924 23:16:15.926601 5611 ts_tablet_manager.cc:1428] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:16:15.926715 5590 heartbeater.cc:503] Master 127.1.4.254:36747 was elected leader, sending a full tablet report...
I20260924 23:16:15.927657 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.924659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:42855 (local address 127.1.4.194:39429)
0924 23:16:15.924694 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:15.924696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:15.924702 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:16:15.924722 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:15.924723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:15.924731 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:15.924783 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:15.925050 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.925665 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.926917 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.927331 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.927429 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.927442 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:15.927562 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 23:16:15.927599 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:16:15.927665 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.924212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:39429 (local address 127.1.4.193:42855)
0924 23:16:15.924395 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:15.924406 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:15.924472 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:15.924898 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:15.924899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:15.924901 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:15.924962 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:15.924963 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.925742 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.925742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:15.926845 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:15.926846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.927428 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.927429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:15.927499 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.927503 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:15.927549 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 23:16:15.927582 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
I20260924 23:16:15.927691 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.923923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:34523 (local address 127.1.4.195:39819)
0924 23:16:15.923959 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:15.923962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:15.924504 (+ 542us) server_negotiation.cc:407] Connection header received
0924 23:16:15.924529 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:15.924530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:15.924542 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:15.924593 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:15.924741 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.925447 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.926865 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.927358 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.927439 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.927449 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:15.927607 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 23:16:15.927642 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 23:16:15.927919 5411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f9d94ea6164d81aa5eaaafef7df359" candidate_uuid: "5793ff0385e2401dae329af7feccf81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffbcfafb94d545e39565faf2ab3f9614" is_pre_election: true
I20260924 23:16:15.928012 5326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.923882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:39819 (local address 127.1.4.193:34523)
0924 23:16:15.924343 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:15.924354 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:15.924403 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:15.924598 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:15.924599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:15.924600 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:15.924665 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:15.924667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.925608 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.925609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:15.926771 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:15.926774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.927453 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.927455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:15.927542 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.927546 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:15.927596 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 23:16:15.927634 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444}
I20260924 23:16:15.928038 5411 raft_consensus.cc:2454] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5793ff0385e2401dae329af7feccf81f in term 0.
I20260924 23:16:15.928228 5211 leader_election.cc:302] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5793ff0385e2401dae329af7feccf81f, ffbcfafb94d545e39565faf2ab3f9614; no voters:
I20260924 23:16:15.928318 5544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f9d94ea6164d81aa5eaaafef7df359" candidate_uuid: "5793ff0385e2401dae329af7feccf81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "091fb4c9d1e64c7d9f5571282dde7fdd" is_pre_election: true
I20260924 23:16:15.928344 5612 raft_consensus.cc:2781] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:15.928390 5612 raft_consensus.cc:482] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:15.928421 5612 raft_consensus.cc:3037] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:15.928431 5544 raft_consensus.cc:2454] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5793ff0385e2401dae329af7feccf81f in term 0.
I20260924 23:16:15.929215 5612 raft_consensus.cc:504] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffbcfafb94d545e39565faf2ab3f9614" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 39429 } } peers { permanent_uuid: "5793ff0385e2401dae329af7feccf81f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36727 } } peers { permanent_uuid: "091fb4c9d1e64c7d9f5571282dde7fdd" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39819 } }
I20260924 23:16:15.929414 5612 leader_election.cc:288] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [CANDIDATE]: Term 1 election: Requested vote from peers ffbcfafb94d545e39565faf2ab3f9614 (127.1.4.194:39429), 091fb4c9d1e64c7d9f5571282dde7fdd (127.1.4.195:39819)
I20260924 23:16:15.929548 5544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f9d94ea6164d81aa5eaaafef7df359" candidate_uuid: "5793ff0385e2401dae329af7feccf81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "091fb4c9d1e64c7d9f5571282dde7fdd"
I20260924 23:16:15.929621 5544 raft_consensus.cc:3037] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:15.929598 5411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f9d94ea6164d81aa5eaaafef7df359" candidate_uuid: "5793ff0385e2401dae329af7feccf81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffbcfafb94d545e39565faf2ab3f9614"
I20260924 23:16:15.929666 5411 raft_consensus.cc:3037] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:15.930233 5544 raft_consensus.cc:2454] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5793ff0385e2401dae329af7feccf81f in term 1.
I20260924 23:16:15.930328 5411 raft_consensus.cc:2454] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5793ff0385e2401dae329af7feccf81f in term 1.
I20260924 23:16:15.930387 5212 leader_election.cc:302] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 091fb4c9d1e64c7d9f5571282dde7fdd, 5793ff0385e2401dae329af7feccf81f; no voters:
I20260924 23:16:15.930495 5612 raft_consensus.cc:2781] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:15.930547 5612 raft_consensus.cc:686] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [term 1 LEADER]: Becoming Leader. State: Replica: 5793ff0385e2401dae329af7feccf81f, State: Running, Role: LEADER
I20260924 23:16:15.930636 5612 consensus_queue.cc:237] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffbcfafb94d545e39565faf2ab3f9614" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 39429 } } peers { permanent_uuid: "5793ff0385e2401dae329af7feccf81f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36727 } } peers { permanent_uuid: "091fb4c9d1e64c7d9f5571282dde7fdd" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39819 } }
I20260924 23:16:15.931425 5134 catalog_manager.cc:5495] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5793ff0385e2401dae329af7feccf81f (127.1.4.193). New cstate: current_term: 1 leader_uuid: "5793ff0385e2401dae329af7feccf81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffbcfafb94d545e39565faf2ab3f9614" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 39429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5793ff0385e2401dae329af7feccf81f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 36727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "091fb4c9d1e64c7d9f5571282dde7fdd" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39819 } health_report { overall_health: UNKNOWN } } }
I20260924 23:16:15.938346 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.936157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.1.4.193:36727)
0924 23:16:15.936204 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:15.936206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:15.936215 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:15.936273 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:15.936274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:15.936290 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:15.936337 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:15.936433 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.936915 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.937682 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:15.937857 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:15.937904 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:15.938063 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:15.938154 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:15.938211 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:15.938223 (+ 12us) server_negotiation.cc:299] Negotiation successful
0924 23:16:15.938238 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:16:15.940037 5411 raft_consensus.cc:1261] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614 [term 1 FOLLOWER]: Refusing update from remote peer 5793ff0385e2401dae329af7feccf81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:15.940078 5544 raft_consensus.cc:1261] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd [term 1 FOLLOWER]: Refusing update from remote peer 5793ff0385e2401dae329af7feccf81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:15.940364 5613 consensus_queue.cc:1035] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffbcfafb94d545e39565faf2ab3f9614" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 39429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:15.940483 5612 consensus_queue.cc:1035] T b0f9d94ea6164d81aa5eaaafef7df359 P 5793ff0385e2401dae329af7feccf81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "091fb4c9d1e64c7d9f5571282dde7fdd" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:15.943140 5622 mvcc.cc:205] Tried to move back new op lower bound from 7333035114247970816 to 7333035114212368384. Current Snapshot: MvccSnapshot[applied={T|T < 7333035114247970816}]
I20260924 23:16:15.943156 5621 mvcc.cc:205] Tried to move back new op lower bound from 7333035114247970816 to 7333035114212368384. Current Snapshot: MvccSnapshot[applied={T|T < 7333035114247970816 or (T in {7333035114247970816})}]
I20260924 23:16:15.945078 5133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:15.945158 5133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:15.945829 5133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 730734a33ba2484e9af312629552226e: Sending DeleteTablet for 3 replicas of tablet b0f9d94ea6164d81aa5eaaafef7df359
I20260924 23:16:15.946171 5523 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f9d94ea6164d81aa5eaaafef7df359 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:15 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:37204
I20260924 23:16:15.946197 5391 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f9d94ea6164d81aa5eaaafef7df359 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:15 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:53744
I20260924 23:16:15.946359 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 5196
I20260924 23:16:15.946379 5629 tablet_replica.cc:330] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd: stopping tablet replica
I20260924 23:16:15.946380 5258 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f9d94ea6164d81aa5eaaafef7df359 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:15 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:44030
I20260924 23:16:15.946995 5629 raft_consensus.cc:2229] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:15.947106 5629 raft_consensus.cc:2258] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:15.947537 5629 ts_tablet_manager.cc:1905] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:15.947839 5630 tablet_replica.cc:330] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614: stopping tablet replica
I20260924 23:16:15.947947 5630 raft_consensus.cc:2229] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:15.948051 5630 raft_consensus.cc:2258] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:15.948472 5630 ts_tablet_manager.cc:1905] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:15.949345 5629 ts_tablet_manager.cc:1918] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:15.949414 5629 log.cc:1196] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/b0f9d94ea6164d81aa5eaaafef7df359
I20260924 23:16:15.949673 5629 ts_tablet_manager.cc:1939] T b0f9d94ea6164d81aa5eaaafef7df359 P 091fb4c9d1e64c7d9f5571282dde7fdd: Deleting consensus metadata
I20260924 23:16:15.949944 5630 ts_tablet_manager.cc:1918] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:15.950026 5630 log.cc:1196] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/b0f9d94ea6164d81aa5eaaafef7df359
I20260924 23:16:15.950063 5119 catalog_manager.cc:4841] TS 091fb4c9d1e64c7d9f5571282dde7fdd (127.1.4.195:39819): tablet b0f9d94ea6164d81aa5eaaafef7df359 (table test-table [id=f8517fb569ba4ad1abc995b8c3d9be23]) successfully deleted
I20260924 23:16:15.950265 5630 ts_tablet_manager.cc:1939] T b0f9d94ea6164d81aa5eaaafef7df359 P ffbcfafb94d545e39565faf2ab3f9614: Deleting consensus metadata
I20260924 23:16:15.950657 5118 catalog_manager.cc:4841] TS ffbcfafb94d545e39565faf2ab3f9614 (127.1.4.194:39429): tablet b0f9d94ea6164d81aa5eaaafef7df359 (table test-table [id=f8517fb569ba4ad1abc995b8c3d9be23]) successfully deleted
W20260924 23:16:15.951952 5119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.4.193:36727 (error 108)
I20260924 23:16:15.952149 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 5329
I20260924 23:16:15.955689 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:15.955536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:36727 (local address 127.0.0.1:44034)
0924 23:16:15.955599 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:15.955653 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:36727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 23:16:15.955838 5119 catalog_manager.cc:4584] TS 5793ff0385e2401dae329af7feccf81f (127.1.4.193:36727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0f9d94ea6164d81aa5eaaafef7df359: Network error: Client connection negotiation failed: client connection to 127.1.4.193:36727: connect: Connection refused (error 111)
I20260924 23:16:15.956949 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 5462
I20260924 23:16:15.960672 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 5103
2026-09-24T23:16:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1772 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T23:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:15Z Disabled control of system clock
I20260924 23:16:15.985620 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:35139
--webserver_interface=127.1.4.254
--webserver_port=0
--builtin_ntp_servers=127.1.4.212:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:35139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 23:16:16.065368 5635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:16.065590 5635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:16.065610 5635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:16.066735 5635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:16.066771 5635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:16.066781 5635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:16.066792 5635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:16.066802 5635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:16.068123 5635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:35139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:35139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:16.068323 5635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:16.068576 5635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:16.073367 5643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:16.073371 5640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:16.073553 5635 server_base.cc:1034] running on GCE node
W20260924 23:16:16.073686 5641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:16.073908 5635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:16.074178 5635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:16.075325 5635 hybrid_clock.cc:648] HybridClock initialized: now 1790291776075305 us; error 34 us; skew 500 ppm
I20260924 23:16:16.076516 5635 webserver.cc:460] Webserver started at http://127.1.4.254:36523/ using document root <none> and password file <none>
I20260924 23:16:16.076731 5635 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:16.076774 5635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:16.076897 5635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:16.078030 5635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "296a15252f0d45cd978aa8ef95be3dca"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "1df2c18fb0d7db3d73e7d829494b7ad6"
server_key_iv: "9d740057137708115cf4d1812be55c95"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.078357 5635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "296a15252f0d45cd978aa8ef95be3dca"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "1df2c18fb0d7db3d73e7d829494b7ad6"
server_key_iv: "9d740057137708115cf4d1812be55c95"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.079640 5635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:16:16.080307 5649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:16.080444 5635 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:16.080521 5635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "296a15252f0d45cd978aa8ef95be3dca"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "1df2c18fb0d7db3d73e7d829494b7ad6"
server_key_iv: "9d740057137708115cf4d1812be55c95"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.080590 5635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:16.083909 5635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:16.084167 5635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:16.084283 5635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:16.087787 5635 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:35139
I20260924 23:16:16.087815 5701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:35139 every 8 connection(s)
I20260924 23:16:16.088127 5635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:16.088737 5702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:16.089437 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 5635
I20260924 23:16:16.089529 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:16.089676 1043 external_mini_cluster.cc:1428] Setting key 37d8eba59afdf11759cdf203636150fc
I20260924 23:16:16.091845 5702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca: Bootstrap starting.
I20260924 23:16:16.092061 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.090152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.1.4.254:35139)
0924 23:16:16.090313 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:16.090320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:16.090344 (+ 24us) server_negotiation.cc:407] Connection header received
0924 23:16:16.090764 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:16.090770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:16.090810 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:16.090922 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:16.091647 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:16.091651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:16.091651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:16.091655 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:16.091693 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:16.091704 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:16.091751 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:16.091821 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 23:16:16.091872 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260924 23:16:16.092504 5702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:16.092768 5702 log.cc:824] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:16.093369 5702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca: No bootstrap required, opened a new log
I20260924 23:16:16.094628 5702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "296a15252f0d45cd978aa8ef95be3dca" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35139 } }
I20260924 23:16:16.094759 5702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:16.094780 5702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296a15252f0d45cd978aa8ef95be3dca, State: Initialized, Role: FOLLOWER
I20260924 23:16:16.094878 5702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "296a15252f0d45cd978aa8ef95be3dca" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35139 } }
I20260924 23:16:16.094961 5702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:16.094986 5702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:16.095062 5702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:16.095896 5702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "296a15252f0d45cd978aa8ef95be3dca" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35139 } }
I20260924 23:16:16.095986 5702 leader_election.cc:302] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 296a15252f0d45cd978aa8ef95be3dca; no voters:
I20260924 23:16:16.096151 5702 leader_election.cc:288] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:16.096190 5707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:16.096297 5707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [term 1 LEADER]: Becoming Leader. State: Replica: 296a15252f0d45cd978aa8ef95be3dca, State: Running, Role: LEADER
I20260924 23:16:16.096377 5707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "296a15252f0d45cd978aa8ef95be3dca" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35139 } }
I20260924 23:16:16.096411 5702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:16.096731 5707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 296a15252f0d45cd978aa8ef95be3dca. Latest consensus state: current_term: 1 leader_uuid: "296a15252f0d45cd978aa8ef95be3dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "296a15252f0d45cd978aa8ef95be3dca" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35139 } } }
I20260924 23:16:16.096803 5707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:16.096762 5709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "296a15252f0d45cd978aa8ef95be3dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "296a15252f0d45cd978aa8ef95be3dca" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35139 } } }
I20260924 23:16:16.096850 5709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:16.096999 5712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:16.097476 5712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:16.098860 5712 catalog_manager.cc:1347] Generated new cluster ID: 07eae2e9344f43e39cf0983bd87ef395
I20260924 23:16:16.098907 5712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:16:16.109512 5712 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:16.109701 5712 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:16.113606 5712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca: Generated new TSK 0
I20260924 23:16:16.113792 5712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:16:16.116145 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:35139
--builtin_ntp_servers=127.1.4.212:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 23:16:16.198531 5726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:16.198738 5726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:16.198768 5726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:16.199982 5726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:16.200032 5726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:16.200069 5726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:16:16.201401 5726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:35139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:16.201646 5726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:16.201934 5726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:16.206907 5734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:16.206981 5731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:16.207115 5726 server_base.cc:1034] running on GCE node
W20260924 23:16:16.207188 5732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:16.207465 5726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:16.207701 5726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:16.208853 5726 hybrid_clock.cc:648] HybridClock initialized: now 1790291776208843 us; error 29 us; skew 500 ppm
I20260924 23:16:16.210120 5726 webserver.cc:460] Webserver started at http://127.1.4.193:46377/ using document root <none> and password file <none>
I20260924 23:16:16.210356 5726 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:16.210403 5726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:16.210530 5726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:16.211647 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "1d9443c3774743088804aa00a8da8711"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "fe7d536b2e94f99e1f51ed2b856da513"
server_key_iv: "eb962c4f2f54a064e82d72245af6001c"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.211997 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "1d9443c3774743088804aa00a8da8711"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "fe7d536b2e94f99e1f51ed2b856da513"
server_key_iv: "eb962c4f2f54a064e82d72245af6001c"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.213193 5726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:16:16.213886 5740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:16.214046 5726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:16.214128 5726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "1d9443c3774743088804aa00a8da8711"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "fe7d536b2e94f99e1f51ed2b856da513"
server_key_iv: "eb962c4f2f54a064e82d72245af6001c"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.214196 5726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:16.217473 5726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:16.217722 5726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:16.217833 5726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:16.218015 5726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:16.218339 5726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:16.218374 5726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:16.218394 5726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:16.218417 5726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:16.224594 5726 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:42071
I20260924 23:16:16.224645 5853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:42071 every 8 connection(s)
I20260924 23:16:16.224960 5726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:16:16.228020 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.225678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:43883 (local address 127.1.4.254:35139)
0924 23:16:16.225760 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:16.225762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:16.225953 (+ 191us) server_negotiation.cc:407] Connection header received
0924 23:16:16.226430 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:16.226431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:16.226437 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:16.226494 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:16.227779 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:16.227780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:16.227781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:16.227784 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:16.227798 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:16.227802 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:16.227850 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:16.227950 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 23:16:16.227963 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 23:16:16.228222 5856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.225653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:35139 (local address 127.1.4.193:43883)
0924 23:16:16.225827 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:16.225863 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:16.226334 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:16.226575 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:16.226579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:16.226587 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:16.227624 (+ 1037us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:16:16.227624 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:16.227656 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:16:16.227687 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:16.227877 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:16.227883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:16.227885 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:16:16.227957 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 23:16:16.228016 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260924 23:16:16.228667 5854 heartbeater.cc:347] Connected to a master server at 127.1.4.254:35139
I20260924 23:16:16.228753 5854 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:16.228888 5854 heartbeater.cc:511] Master 127.1.4.254:35139 requested a full tablet report, sending...
I20260924 23:16:16.229331 5666 ts_manager.cc:194] Registered new tserver with Master: 1d9443c3774743088804aa00a8da8711 (127.1.4.193:42071)
I20260924 23:16:16.232158 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 5726
I20260924 23:16:16.232252 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:16:16.232406 1043 external_mini_cluster.cc:1428] Setting key d457794104bed3b4357bc701af478f39
I20260924 23:16:16.233299 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:35139
--builtin_ntp_servers=127.1.4.212:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 23:16:16.312191 5857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:16.312402 5857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:16.312423 5857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:16.313576 5857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:16.313618 5857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:16.313645 5857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:16:16.314968 5857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:35139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:16.315205 5857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:16.315500 5857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:16.320444 5865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:16.320426 5857 server_base.cc:1034] running on GCE node
W20260924 23:16:16.320550 5863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:16.320407 5862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:16.320842 5857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:16.321082 5857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:16.322221 5857 hybrid_clock.cc:648] HybridClock initialized: now 1790291776322210 us; error 27 us; skew 500 ppm
I20260924 23:16:16.323603 5857 webserver.cc:460] Webserver started at http://127.1.4.194:45641/ using document root <none> and password file <none>
I20260924 23:16:16.323817 5857 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:16.323859 5857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:16.323947 5857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:16.324954 5857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "19723a407d1643c7a9cd5a3dbc1fda04"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "0017244741a6f00789be69447236dbf4"
server_key_iv: "96a29e75cda767490e1faba291e1e0ec"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.325258 5857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "19723a407d1643c7a9cd5a3dbc1fda04"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "0017244741a6f00789be69447236dbf4"
server_key_iv: "96a29e75cda767490e1faba291e1e0ec"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.326467 5857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:16.327143 5871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:16.327320 5857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:16.327421 5857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "19723a407d1643c7a9cd5a3dbc1fda04"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "0017244741a6f00789be69447236dbf4"
server_key_iv: "96a29e75cda767490e1faba291e1e0ec"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.327498 5857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:16.330814 5857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:16.331061 5857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:16.331187 5857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:16.331398 5857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:16.331701 5857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:16.331733 5857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:16.331753 5857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:16.331774 5857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:16.337070 5857 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:35819
I20260924 23:16:16.337141 5984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:35819 every 8 connection(s)
I20260924 23:16:16.337426 5857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:16:16.340266 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.338109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:41301 (local address 127.1.4.254:35139)
0924 23:16:16.338183 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:16.338185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:16.338384 (+ 199us) server_negotiation.cc:407] Connection header received
0924 23:16:16.338834 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:16.338835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:16.338842 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:16.338895 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:16.340041 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:16.340043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:16.340043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:16.340047 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:16.340063 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:16.340066 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:16.340122 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:16.340185 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 23:16:16.340204 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 23:16:16.340425 5987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.338084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:35139 (local address 127.1.4.194:41301)
0924 23:16:16.338269 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:16.338304 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:16.338754 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:16.338953 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:16.338957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:16.338965 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:16.339860 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:16:16.339860 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:16.339890 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:16:16.339920 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:16.340126 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:16.340129 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:16.340132 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:16.340187 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 23:16:16.340237 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260924 23:16:16.340865 5985 heartbeater.cc:347] Connected to a master server at 127.1.4.254:35139
I20260924 23:16:16.340942 5985 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:16.341064 5985 heartbeater.cc:511] Master 127.1.4.254:35139 requested a full tablet report, sending...
I20260924 23:16:16.341362 5666 ts_manager.cc:194] Registered new tserver with Master: 19723a407d1643c7a9cd5a3dbc1fda04 (127.1.4.194:35819)
I20260924 23:16:16.347596 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 5857
I20260924 23:16:16.347688 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:16:16.347820 1043 external_mini_cluster.cc:1428] Setting key 2a3d0e6d6b8cda2da394436e581cf1de
I20260924 23:16:16.348788 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:35139
--builtin_ntp_servers=127.1.4.212:40935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 23:16:16.428673 5988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:16.428894 5988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:16.428918 5988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:16.430163 5988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:16.430207 5988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:16.430233 5988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:16:16.431725 5988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:35139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:16.431951 5988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:16.432212 5988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:16.437455 5996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:16.437448 5993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:16.437455 5994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:16.437680 5988 server_base.cc:1034] running on GCE node
I20260924 23:16:16.437911 5988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:16.438167 5988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:16.439328 5988 hybrid_clock.cc:648] HybridClock initialized: now 1790291776439314 us; error 33 us; skew 500 ppm
I20260924 23:16:16.440567 5988 webserver.cc:460] Webserver started at http://127.1.4.195:36349/ using document root <none> and password file <none>
I20260924 23:16:16.440798 5988 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:16.440840 5988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:16.441039 5988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:16.442114 5988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "430985d676cf4886ab350b73e215e9a1"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "6991e3feef9fcb04d9add103af69896a"
server_key_iv: "5c7a09df2e8ee56ccec8607336661595"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.442443 5988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "430985d676cf4886ab350b73e215e9a1"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "6991e3feef9fcb04d9add103af69896a"
server_key_iv: "5c7a09df2e8ee56ccec8607336661595"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.443707 5988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:16.444433 6002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:16.444589 5988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:16.444672 5988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "430985d676cf4886ab350b73e215e9a1"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "6991e3feef9fcb04d9add103af69896a"
server_key_iv: "5c7a09df2e8ee56ccec8607336661595"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.444733 5988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:16.448122 5988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:16.448376 5988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:16.448494 5988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:16.448681 5988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:16.448985 5988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:16.449018 5988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:16.449043 5988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:16.449067 5988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:16.454363 5988 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:32857
I20260924 23:16:16.454423 6115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:32857 every 8 connection(s)
I20260924 23:16:16.454730 5988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:16:16.457690 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.455433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:57699 (local address 127.1.4.254:35139)
0924 23:16:16.455488 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:16.455490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:16.455696 (+ 206us) server_negotiation.cc:407] Connection header received
0924 23:16:16.456212 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:16.456214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:16.456221 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:16.456287 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:16.457479 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:16.457480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:16.457480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:16.457485 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:16.457499 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:16.457504 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:16.457556 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:16.457623 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 23:16:16.457636 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:16:16.457844 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.455416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:35139 (local address 127.1.4.195:57699)
0924 23:16:16.455592 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:16.455626 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:16.456097 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:16.456349 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:16.456353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:16.456361 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:16.457298 (+ 937us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:16:16.457298 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:16.457328 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:16:16.457370 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:16.457565 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:16.457568 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:16.457570 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:16:16.457624 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 23:16:16.457676 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260924 23:16:16.458397 6116 heartbeater.cc:347] Connected to a master server at 127.1.4.254:35139
I20260924 23:16:16.458479 6116 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:16.458608 6116 heartbeater.cc:511] Master 127.1.4.254:35139 requested a full tablet report, sending...
I20260924 23:16:16.459003 5666 ts_manager.cc:194] Registered new tserver with Master: 430985d676cf4886ab350b73e215e9a1 (127.1.4.195:32857)
I20260924 23:16:16.463263 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 5988
I20260924 23:16:16.463344 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:16:16.463526 1043 external_mini_cluster.cc:1428] Setting key 43bbc9d4c5b5e12ef387fb298543a340
I20260924 23:16:16.464259 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:16:16.466588 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.465591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.1.4.254:35139)
0924 23:16:16.465647 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:16.465649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:16.465738 (+ 89us) server_negotiation.cc:407] Connection header received
0924 23:16:16.465793 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:16.465794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:16.465799 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:16.465840 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:16.466364 (+ 524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:16.466365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:16.466365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:16.466370 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:16.466383 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:16.466387 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:16.466434 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:16.466480 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 23:16:16.466494 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 23:16:16.468149 5666 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:16:16.469620 5666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:16.473345 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.471552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:35819 (local address 127.0.0.1:36336)
0924 23:16:16.471674 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:16.471712 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:16.471762 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:16.472057 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:16.472061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:16.472069 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:16.472533 (+ 464us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:16:16.472533 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:16.472548 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:16:16.472570 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:16.473155 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:16.473160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:16.473164 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:16.473207 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 23:16:16.473220 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260924 23:16:16.473467 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.471897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:42071 (local address 127.0.0.1:38780)
0924 23:16:16.472282 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:16.472290 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:16.472322 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:16.472465 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:16.472465 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:16.472467 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:16.473105 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:16:16.473106 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:16.473111 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:16:16.473120 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:16.473357 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:16.473358 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:16.473359 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 23:16:16.473392 (+ 33us) client_negotiation.cc:240] Negotiation successful
0924 23:16:16.473402 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":63,"threads_started":1}
I20260924 23:16:16.473389 6125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.471657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.1.4.194:35819)
0924 23:16:16.471847 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:16.471852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:16.471863 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:16:16.471888 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:16.471891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:16.471902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:16.472053 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:16.472632 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:16.472637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:16.472637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:16.472640 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:16.473045 (+ 405us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:16.473070 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:16.473271 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:16.473294 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 23:16:16.473308 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260924 23:16:16.473654 6129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.472130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38780 (local address 127.1.4.193:42071)
0924 23:16:16.472311 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:16.472315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:16.472326 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:16:16.472375 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:16.472377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:16.472386 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:16.472458 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:16.473211 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:16.473219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:16.473219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:16.473223 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:16.473256 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:16.473282 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:16.473318 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:16.473549 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 23:16:16.473566 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260924 23:16:16.473722 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.471748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:32857 (local address 127.0.0.1:35528)
0924 23:16:16.471857 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:16.471865 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:16.471895 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:16.472545 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:16.472546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:16.472547 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:16.472883 (+ 336us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:16:16.472884 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:16.472889 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:16:16.472900 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:16.473155 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:16.473160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:16.473163 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:16.473223 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 23:16:16.473235 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260924 23:16:16.473928 6127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.471839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.1.4.195:32857)
0924 23:16:16.472156 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:16.472161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:16.472170 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:16.472194 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:16.472196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:16.472205 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:16.472273 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:16.472978 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:16.472986 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:16.472986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:16.472990 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:16.473056 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:16.473080 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:16.473132 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:16.473309 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 23:16:16.473324 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":75,"threads_started":1}
I20260924 23:16:16.474272 5788 tablet_service.cc:1467] Processing CreateTablet for tablet fccd9ec2964e469abc41670fafed99d5 (DEFAULT_TABLE table=test-table [id=8a8823b875be43a5b753133eae6788a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:16.474256 5919 tablet_service.cc:1467] Processing CreateTablet for tablet fccd9ec2964e469abc41670fafed99d5 (DEFAULT_TABLE table=test-table [id=8a8823b875be43a5b753133eae6788a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:16.474592 5788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fccd9ec2964e469abc41670fafed99d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:16.474594 5919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fccd9ec2964e469abc41670fafed99d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:16.474535 6050 tablet_service.cc:1467] Processing CreateTablet for tablet fccd9ec2964e469abc41670fafed99d5 (DEFAULT_TABLE table=test-table [id=8a8823b875be43a5b753133eae6788a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:16.474777 6050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fccd9ec2964e469abc41670fafed99d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:16.477726 6131 tablet_bootstrap.cc:492] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1: Bootstrap starting.
I20260924 23:16:16.477999 6132 tablet_bootstrap.cc:492] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04: Bootstrap starting.
I20260924 23:16:16.478128 6130 tablet_bootstrap.cc:492] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711: Bootstrap starting.
I20260924 23:16:16.478418 6131 tablet_bootstrap.cc:654] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:16.478638 6132 tablet_bootstrap.cc:654] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:16.478677 6130 tablet_bootstrap.cc:654] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:16.478685 6131 log.cc:824] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:16.478947 6130 log.cc:824] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:16.478947 6132 log.cc:824] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:16.479442 6131 tablet_bootstrap.cc:492] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1: No bootstrap required, opened a new log
I20260924 23:16:16.479521 6131 ts_tablet_manager.cc:1397] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:16:16.479642 6130 tablet_bootstrap.cc:492] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711: No bootstrap required, opened a new log
I20260924 23:16:16.479663 6132 tablet_bootstrap.cc:492] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04: No bootstrap required, opened a new log
I20260924 23:16:16.479698 6130 ts_tablet_manager.cc:1397] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:16.479720 6132 ts_tablet_manager.cc:1397] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:16.481485 6132 raft_consensus.cc:348] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19723a407d1643c7a9cd5a3dbc1fda04" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 35819 } } peers { permanent_uuid: "430985d676cf4886ab350b73e215e9a1" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 32857 } } peers { permanent_uuid: "1d9443c3774743088804aa00a8da8711" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42071 } }
I20260924 23:16:16.481626 6132 raft_consensus.cc:374] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:16.481655 6132 raft_consensus.cc:729] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19723a407d1643c7a9cd5a3dbc1fda04, State: Initialized, Role: FOLLOWER
I20260924 23:16:16.481766 6132 consensus_queue.cc:260] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19723a407d1643c7a9cd5a3dbc1fda04" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 35819 } } peers { permanent_uuid: "430985d676cf4886ab350b73e215e9a1" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 32857 } } peers { permanent_uuid: "1d9443c3774743088804aa00a8da8711" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42071 } }
I20260924 23:16:16.482010 6132 ts_tablet_manager.cc:1428] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 23:16:16.482021 5985 heartbeater.cc:503] Master 127.1.4.254:35139 was elected leader, sending a full tablet report...
I20260924 23:16:16.482151 6131 raft_consensus.cc:348] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19723a407d1643c7a9cd5a3dbc1fda04" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 35819 } } peers { permanent_uuid: "430985d676cf4886ab350b73e215e9a1" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 32857 } } peers { permanent_uuid: "1d9443c3774743088804aa00a8da8711" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42071 } }
I20260924 23:16:16.482151 6130 raft_consensus.cc:348] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19723a407d1643c7a9cd5a3dbc1fda04" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 35819 } } peers { permanent_uuid: "430985d676cf4886ab350b73e215e9a1" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 32857 } } peers { permanent_uuid: "1d9443c3774743088804aa00a8da8711" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42071 } }
I20260924 23:16:16.482306 6131 raft_consensus.cc:374] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:16.482306 6130 raft_consensus.cc:374] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:16.482344 6131 raft_consensus.cc:729] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 430985d676cf4886ab350b73e215e9a1, State: Initialized, Role: FOLLOWER
I20260924 23:16:16.482344 6130 raft_consensus.cc:729] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9443c3774743088804aa00a8da8711, State: Initialized, Role: FOLLOWER
I20260924 23:16:16.482450 6130 consensus_queue.cc:260] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19723a407d1643c7a9cd5a3dbc1fda04" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 35819 } } peers { permanent_uuid: "430985d676cf4886ab350b73e215e9a1" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 32857 } } peers { permanent_uuid: "1d9443c3774743088804aa00a8da8711" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42071 } }
I20260924 23:16:16.482463 6131 consensus_queue.cc:260] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19723a407d1643c7a9cd5a3dbc1fda04" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 35819 } } peers { permanent_uuid: "430985d676cf4886ab350b73e215e9a1" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 32857 } } peers { permanent_uuid: "1d9443c3774743088804aa00a8da8711" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42071 } }
I20260924 23:16:16.482694 6130 ts_tablet_manager.cc:1428] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 23:16:16.482694 6131 ts_tablet_manager.cc:1428] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:16:16.482796 5854 heartbeater.cc:503] Master 127.1.4.254:35139 was elected leader, sending a full tablet report...
I20260924 23:16:16.483137 6116 heartbeater.cc:503] Master 127.1.4.254:35139 was elected leader, sending a full tablet report...
I20260924 23:16:16.560712 6136 raft_consensus.cc:482] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:16.560837 6136 raft_consensus.cc:504] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19723a407d1643c7a9cd5a3dbc1fda04" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 35819 } } peers { permanent_uuid: "430985d676cf4886ab350b73e215e9a1" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 32857 } } peers { permanent_uuid: "1d9443c3774743088804aa00a8da8711" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42071 } }
I20260924 23:16:16.561200 6136 leader_election.cc:288] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 430985d676cf4886ab350b73e215e9a1 (127.1.4.195:32857), 1d9443c3774743088804aa00a8da8711 (127.1.4.193:42071)
I20260924 23:16:16.562451 6127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.561490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:36311 (local address 127.1.4.195:32857)
0924 23:16:16.561547 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:16.561549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:16.561563 (+ 14us) server_negotiation.cc:407] Connection header received
0924 23:16:16.561593 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:16.561594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:16.561600 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:16.561670 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:16.562235 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:16.562237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:16.562237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:16.562243 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:16.562262 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:16.562267 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:16.562326 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:16.562375 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 23:16:16.562388 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 23:16:16.562451 5987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.561418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:32857 (local address 127.1.4.194:36311)
0924 23:16:16.561478 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:16.561490 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:16.561531 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:16.561680 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:16.561681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:16.561684 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:16.562128 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:16:16.562128 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:16.562136 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:16:16.562150 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:16.562324 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:16.562325 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:16.562325 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 23:16:16.562374 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 23:16:16.562387 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 23:16:16.562657 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.561533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:42071 (local address 127.1.4.194:42025)
0924 23:16:16.561716 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:16.561724 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:16.561769 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:16.561884 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:16.561885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:16.561899 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:16.562275 (+ 376us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:16:16.562275 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:16.562281 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:16:16.562292 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:16.562450 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:16.562451 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:16.562490 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 23:16:16.562568 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 23:16:16.562580 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":113,"threads_started":1}
I20260924 23:16:16.562654 6129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.561624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:42025 (local address 127.1.4.193:42071)
0924 23:16:16.561688 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:16.561690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:16.561774 (+ 84us) server_negotiation.cc:407] Connection header received
0924 23:16:16.561817 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:16.561818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:16.561822 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:16.561868 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:16.562367 (+ 499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:16.562368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:16.562368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:16.562374 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:16.562391 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:16.562396 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:16.562447 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:16.562580 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 23:16:16.562594 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 23:16:16.562829 6070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccd9ec2964e469abc41670fafed99d5" candidate_uuid: "19723a407d1643c7a9cd5a3dbc1fda04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "430985d676cf4886ab350b73e215e9a1" is_pre_election: true
I20260924 23:16:16.562980 6070 raft_consensus.cc:2454] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19723a407d1643c7a9cd5a3dbc1fda04 in term 0.
I20260924 23:16:16.563218 5873 leader_election.cc:302] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19723a407d1643c7a9cd5a3dbc1fda04, 430985d676cf4886ab350b73e215e9a1; no voters:
I20260924 23:16:16.563344 6136 raft_consensus.cc:2781] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:16.563316 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccd9ec2964e469abc41670fafed99d5" candidate_uuid: "19723a407d1643c7a9cd5a3dbc1fda04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9443c3774743088804aa00a8da8711" is_pre_election: true
I20260924 23:16:16.563412 6136 raft_consensus.cc:482] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:16.563444 6136 raft_consensus.cc:3037] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:16.563477 5808 raft_consensus.cc:2454] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19723a407d1643c7a9cd5a3dbc1fda04 in term 0.
I20260924 23:16:16.564217 6136 raft_consensus.cc:504] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19723a407d1643c7a9cd5a3dbc1fda04" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 35819 } } peers { permanent_uuid: "430985d676cf4886ab350b73e215e9a1" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 32857 } } peers { permanent_uuid: "1d9443c3774743088804aa00a8da8711" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42071 } }
I20260924 23:16:16.564334 6136 leader_election.cc:288] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [CANDIDATE]: Term 1 election: Requested vote from peers 430985d676cf4886ab350b73e215e9a1 (127.1.4.195:32857), 1d9443c3774743088804aa00a8da8711 (127.1.4.193:42071)
I20260924 23:16:16.564519 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccd9ec2964e469abc41670fafed99d5" candidate_uuid: "19723a407d1643c7a9cd5a3dbc1fda04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9443c3774743088804aa00a8da8711"
I20260924 23:16:16.564539 6070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fccd9ec2964e469abc41670fafed99d5" candidate_uuid: "19723a407d1643c7a9cd5a3dbc1fda04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "430985d676cf4886ab350b73e215e9a1"
I20260924 23:16:16.564594 5808 raft_consensus.cc:3037] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:16.564601 6070 raft_consensus.cc:3037] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:16.565305 6070 raft_consensus.cc:2454] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19723a407d1643c7a9cd5a3dbc1fda04 in term 1.
I20260924 23:16:16.565348 5808 raft_consensus.cc:2454] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19723a407d1643c7a9cd5a3dbc1fda04 in term 1.
I20260924 23:16:16.565498 5873 leader_election.cc:302] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19723a407d1643c7a9cd5a3dbc1fda04, 1d9443c3774743088804aa00a8da8711; no voters:
I20260924 23:16:16.565593 6136 raft_consensus.cc:2781] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:16.565726 6136 raft_consensus.cc:686] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [term 1 LEADER]: Becoming Leader. State: Replica: 19723a407d1643c7a9cd5a3dbc1fda04, State: Running, Role: LEADER
I20260924 23:16:16.565802 6136 consensus_queue.cc:237] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19723a407d1643c7a9cd5a3dbc1fda04" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 35819 } } peers { permanent_uuid: "430985d676cf4886ab350b73e215e9a1" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 32857 } } peers { permanent_uuid: "1d9443c3774743088804aa00a8da8711" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42071 } }
I20260924 23:16:16.566570 5665 catalog_manager.cc:5495] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19723a407d1643c7a9cd5a3dbc1fda04 (127.1.4.194). New cstate: current_term: 1 leader_uuid: "19723a407d1643c7a9cd5a3dbc1fda04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19723a407d1643c7a9cd5a3dbc1fda04" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 35819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "430985d676cf4886ab350b73e215e9a1" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 32857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d9443c3774743088804aa00a8da8711" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42071 } health_report { overall_health: UNKNOWN } } }
W20260924 23:16:16.587908 5986 tablet.cc:2369] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:16:16.592540 6125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.591673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.1.4.194:35819)
0924 23:16:16.591723 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:16.591725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:16.591733 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:16.591774 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:16.591775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:16.591787 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:16.591834 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:16.592334 (+ 500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:16.592336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:16.592336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:16.592340 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:16.592354 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:16.592358 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:16.592404 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:16.592454 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 23:16:16.592466 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:16:16.594174 6070 raft_consensus.cc:1261] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1 [term 1 FOLLOWER]: Refusing update from remote peer 19723a407d1643c7a9cd5a3dbc1fda04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:16.594183 5808 raft_consensus.cc:1261] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711 [term 1 FOLLOWER]: Refusing update from remote peer 19723a407d1643c7a9cd5a3dbc1fda04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:16.594502 6140 consensus_queue.cc:1035] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "430985d676cf4886ab350b73e215e9a1" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 32857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:16.594605 6136 consensus_queue.cc:1035] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9443c3774743088804aa00a8da8711" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:16.596848 6143 mvcc.cc:205] Tried to move back new op lower bound from 7333035116927324160 to 7333035116814123008. Current Snapshot: MvccSnapshot[applied={T|T < 7333035116927324160}]
I20260924 23:16:16.597571 6144 mvcc.cc:205] Tried to move back new op lower bound from 7333035116927324160 to 7333035116814123008. Current Snapshot: MvccSnapshot[applied={T|T < 7333035116927324160}]
I20260924 23:16:16.599210 5665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:16.599303 5665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:16.600083 5665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 296a15252f0d45cd978aa8ef95be3dca: Sending DeleteTablet for 3 replicas of tablet fccd9ec2964e469abc41670fafed99d5
I20260924 23:16:16.600363 5919 tablet_service.cc:1514] Processing DeleteTablet for tablet fccd9ec2964e469abc41670fafed99d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:16 UTC) from {username='slave'} at 127.0.0.1:36336
I20260924 23:16:16.600400 6050 tablet_service.cc:1514] Processing DeleteTablet for tablet fccd9ec2964e469abc41670fafed99d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:16 UTC) from {username='slave'} at 127.0.0.1:35528
I20260924 23:16:16.600433 5788 tablet_service.cc:1514] Processing DeleteTablet for tablet fccd9ec2964e469abc41670fafed99d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:16 UTC) from {username='slave'} at 127.0.0.1:38780
I20260924 23:16:16.600623 6155 tablet_replica.cc:330] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711: stopping tablet replica
I20260924 23:16:16.600715 6155 raft_consensus.cc:2229] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:16.600787 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 5726
I20260924 23:16:16.600798 6154 tablet_replica.cc:330] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1: stopping tablet replica
I20260924 23:16:16.600821 6155 raft_consensus.cc:2258] T fccd9ec2964e469abc41670fafed99d5 P 1d9443c3774743088804aa00a8da8711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:16.602536 6154 raft_consensus.cc:2229] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:16.602644 6154 raft_consensus.cc:2258] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:16.603003 6154 ts_tablet_manager.cc:1905] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:16.603482 6153 tablet_replica.cc:330] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04: stopping tablet replica
I20260924 23:16:16.603615 6153 raft_consensus.cc:2229] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:16:16.603808 6153 raft_consensus.cc:2258] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:16.604278 6153 ts_tablet_manager.cc:1905] T fccd9ec2964e469abc41670fafed99d5 P 19723a407d1643c7a9cd5a3dbc1fda04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:16.604691 6154 ts_tablet_manager.cc:1918] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:16.604764 6154 log.cc:1196] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/fccd9ec2964e469abc41670fafed99d5
I20260924 23:16:16.605000 6154 ts_tablet_manager.cc:1939] T fccd9ec2964e469abc41670fafed99d5 P 430985d676cf4886ab350b73e215e9a1: Deleting consensus metadata
I20260924 23:16:16.605441 5651 catalog_manager.cc:4841] TS 430985d676cf4886ab350b73e215e9a1 (127.1.4.195:32857): tablet fccd9ec2964e469abc41670fafed99d5 (table test-table [id=8a8823b875be43a5b753133eae6788a9]) successfully deleted
W20260924 23:16:16.605809 5651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.4.193:42071 (error 108)
I20260924 23:16:16.606053 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 5857
I20260924 23:16:16.606783 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.606668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:42071 (local address 127.0.0.1:38796)
0924 23:16:16.606708 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:16.606748 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:42071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 23:16:16.606889 5651 catalog_manager.cc:4584] TS 1d9443c3774743088804aa00a8da8711 (127.1.4.193:42071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fccd9ec2964e469abc41670fafed99d5: Network error: Client connection negotiation failed: client connection to 127.1.4.193:42071: connect: Connection refused (error 111)
I20260924 23:16:16.610574 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 5988
I20260924 23:16:16.610617 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.610501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:35819 (local address 127.0.0.1:36358)
0924 23:16:16.610544 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:16.610571 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.194:35819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 23:16:16.614374 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 5635
2026-09-24T23:16:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (652 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 23:16:16.632434 1043 mini_oidc.cc:152] Starting JWKS server
I20260924 23:16:16.633442 1043 webserver.cc:460] Webserver started at https://127.0.0.1:38275/ using document root <none> and password file <none>
I20260924 23:16:16.633526 1043 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 23:16:16.633657 1043 webserver.cc:460] Webserver started at http://127.0.0.1:35013/ using document root <none> and password file <none>
2026-09-24T23:16:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:16Z Disabled control of system clock
I20260924 23:16:16.640970 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:34077
--webserver_interface=127.1.4.254
--webserver_port=0
--builtin_ntp_servers=127.1.4.212:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:34077
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38275/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/testchainca.cert with env {}
W20260924 23:16:16.719910 6162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:16.720124 6162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:16.720144 6162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:16.721288 6162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:16.721324 6162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:16.721336 6162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:16.721346 6162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:16.721357 6162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 23:16:16.721369 6162 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 23:16:16.721377 6162 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38275/jwks.json
I20260924 23:16:16.722855 6162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:34077
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:34077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38275/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:16.723062 6162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:16.723227 6162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:16.727964 6170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:16.728017 6162 server_base.cc:1034] running on GCE node
W20260924 23:16:16.727955 6167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:16.727968 6168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:16.728394 6162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:16.728686 6162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:16.729890 6162 hybrid_clock.cc:648] HybridClock initialized: now 1790291776729850 us; error 54 us; skew 500 ppm
I20260924 23:16:16.731173 6162 webserver.cc:460] Webserver started at http://127.1.4.254:43961/ using document root <none> and password file <none>
I20260924 23:16:16.731431 6162 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:16.731506 6162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:16.731626 6162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:16.732636 6162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "a862ff16f6ac40458573bfe85f914c41"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "2a92d92fe0a8e107c0ac106f09a631df"
server_key_iv: "ec19c22e9fb86aa5bc6e66377e1c5275"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.732935 6162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "a862ff16f6ac40458573bfe85f914c41"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "2a92d92fe0a8e107c0ac106f09a631df"
server_key_iv: "ec19c22e9fb86aa5bc6e66377e1c5275"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.734153 6162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:16.735105 6176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:16.735303 6162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:16.735419 6162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "a862ff16f6ac40458573bfe85f914c41"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "2a92d92fe0a8e107c0ac106f09a631df"
server_key_iv: "ec19c22e9fb86aa5bc6e66377e1c5275"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.735483 6162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:16.755965 6162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:16.756263 6162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:16.756361 6162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:16.759970 6162 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:34077
I20260924 23:16:16.760015 6228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:34077 every 8 connection(s)
I20260924 23:16:16.760327 6162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:16.761216 6229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:16.763484 6229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41: Bootstrap starting.
I20260924 23:16:16.764029 6229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:16.764278 6229 log.cc:824] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:16.764950 6229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41: No bootstrap required, opened a new log
I20260924 23:16:16.764925 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 6162
I20260924 23:16:16.765033 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:16.765223 1043 external_mini_cluster.cc:1428] Setting key 00b8f305ca82cb2dea863a45238c1bf5
I20260924 23:16:16.766804 6229 raft_consensus.cc:348] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a862ff16f6ac40458573bfe85f914c41" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 34077 } }
I20260924 23:16:16.766944 6229 raft_consensus.cc:374] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:16.766968 6229 raft_consensus.cc:729] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a862ff16f6ac40458573bfe85f914c41, State: Initialized, Role: FOLLOWER
I20260924 23:16:16.767112 6229 consensus_queue.cc:260] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a862ff16f6ac40458573bfe85f914c41" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 34077 } }
I20260924 23:16:16.767196 6229 raft_consensus.cc:388] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:16.767230 6229 raft_consensus.cc:482] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:16.767278 6229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:16.767879 6229 raft_consensus.cc:504] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a862ff16f6ac40458573bfe85f914c41" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 34077 } }
I20260924 23:16:16.767974 6229 leader_election.cc:302] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a862ff16f6ac40458573bfe85f914c41; no voters:
I20260924 23:16:16.768126 6229 leader_election.cc:288] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:16.768162 6234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:16.768257 6234 raft_consensus.cc:686] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [term 1 LEADER]: Becoming Leader. State: Replica: a862ff16f6ac40458573bfe85f914c41, State: Running, Role: LEADER
I20260924 23:16:16.768335 6234 consensus_queue.cc:237] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a862ff16f6ac40458573bfe85f914c41" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 34077 } }
I20260924 23:16:16.768380 6229 sys_catalog.cc:564] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:16.768651 6234 sys_catalog.cc:455] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a862ff16f6ac40458573bfe85f914c41. Latest consensus state: current_term: 1 leader_uuid: "a862ff16f6ac40458573bfe85f914c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a862ff16f6ac40458573bfe85f914c41" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 34077 } } }
I20260924 23:16:16.768819 6234 sys_catalog.cc:458] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:16.768666 6235 sys_catalog.cc:455] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a862ff16f6ac40458573bfe85f914c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a862ff16f6ac40458573bfe85f914c41" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 34077 } } }
I20260924 23:16:16.769110 6235 sys_catalog.cc:458] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:16.769270 6242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:16.769778 6242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:16.772403 6242 catalog_manager.cc:1347] Generated new cluster ID: 8f9e95ba0e69482898540be5399a0af1
I20260924 23:16:16.772408 6233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.765738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.1.4.254:34077)
0924 23:16:16.765921 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:16.765928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:16.765953 (+ 25us) server_negotiation.cc:407] Connection header received
0924 23:16:16.766783 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:16.766791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:16.766870 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:16.767032 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:16.767273 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:16.768911 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:16.770046 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:16.770668 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:16.770786 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:16.771402 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:16.771406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:16.771406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:16.771606 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:16.771647 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:16.771662 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:16.771731 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:16.772125 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 23:16:16.772203 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":58,"threads_started":1}
I20260924 23:16:16.772464 6242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:16:16.776266 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:16:16.779241 6242 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:16.779750 6242 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:16.791110 6242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a862ff16f6ac40458573bfe85f914c41: Generated new TSK 0
I20260924 23:16:16.791299 6242 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 23:16:16.814339 6259 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 23:16:16.814464 6259 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260924 23:16:16.814944 6258 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:16.800011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:34077 (local address 127.0.0.1:35298)
0924 23:16:16.800170 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:16.800180 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:16.800244 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:16.800404 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:16.800405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:16.800407 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 23:16:16.800481 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:16.800483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:16.801097 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:16.801098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:16.801683 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:16.801685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:16.801994 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:16.801997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:16.802097 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:16.802116 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 23:16:16.814626 (+ 12510us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 23:16:16.814707 (+ 81us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.4.254:34077: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.4.254:34077: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 23:16:16.815248 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 6162
I20260924 23:16:16.815245 6233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.800068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35298 (local address 127.1.4.254:34077)
0924 23:16:16.800110 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:16.800112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:16.800233 (+ 121us) server_negotiation.cc:407] Connection header received
0924 23:16:16.800323 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:16.800324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:16.800337 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:16.800392 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0924 23:16:16.800540 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:16.801026 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:16.801753 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:16.801937 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:16.801990 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:16.802218 (+ 228us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 23:16:16.814455 (+ 12237us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 23:16:16.815186 (+ 731us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35298: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 23:16:16.815287 6233 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35298: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
2026-09-24T23:16:16Z chronyd exiting
I20260924 23:16:16.850558 1043 test_util.cc:175] -----------------------------------------------
I20260924 23:16:16.850656 1043 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291755455739-1043-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (219 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 23:16:16.851441 1043 mini_oidc.cc:152] Starting JWKS server
I20260924 23:16:16.852105 1043 webserver.cc:460] Webserver started at https://127.0.0.1:44047/ using document root <none> and password file <none>
I20260924 23:16:16.852147 1043 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 23:16:16.852273 1043 webserver.cc:460] Webserver started at http://127.0.0.1:42361/ using document root <none> and password file <none>
2026-09-24T23:16:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:16Z Disabled control of system clock
I20260924 23:16:16.860085 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:35949
--webserver_interface=127.1.4.254
--webserver_port=0
--builtin_ntp_servers=127.1.4.212:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:35949
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44047/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/testchainca.cert with env {}
W20260924 23:16:16.940897 6265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:16.941131 6265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:16.941152 6265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:16.942353 6265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:16.942401 6265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:16.942413 6265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:16.942425 6265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:16.942435 6265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 23:16:16.942446 6265 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 23:16:16.942471 6265 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44047/jwks.json
I20260924 23:16:16.944131 6265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:35949
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:35949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44047/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:16.944375 6265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:16.944588 6265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:16.949339 6273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:16.949360 6271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:16.949404 6265 server_base.cc:1034] running on GCE node
W20260924 23:16:16.949371 6270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:16.949756 6265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:16.950101 6265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:16.951273 6265 hybrid_clock.cc:648] HybridClock initialized: now 1790291776951248 us; error 34 us; skew 500 ppm
I20260924 23:16:16.952562 6265 webserver.cc:460] Webserver started at http://127.1.4.254:33007/ using document root <none> and password file <none>
I20260924 23:16:16.952797 6265 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:16.952848 6265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:16.953063 6265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:16.954160 6265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "6f885c67d2b04627a2522a87d6c73897"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "ace4c1416233026d91749d513912ddef"
server_key_iv: "7bb9799f3edff10322d7020bd03f778e"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.954490 6265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "6f885c67d2b04627a2522a87d6c73897"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "ace4c1416233026d91749d513912ddef"
server_key_iv: "7bb9799f3edff10322d7020bd03f778e"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.955747 6265 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:16.956492 6279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:16.956645 6265 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:16.956724 6265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "6f885c67d2b04627a2522a87d6c73897"
format_stamp: "Formatted at 2026-09-24 23:16:16 on dist-test-slave-s1hw"
server_key: "ace4c1416233026d91749d513912ddef"
server_key_iv: "7bb9799f3edff10322d7020bd03f778e"
server_key_version: "encryptionkey@0"
I20260924 23:16:16.956784 6265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:16.972501 6265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:16.972857 6265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:16.973009 6265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:16.977205 6265 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:35949
I20260924 23:16:16.977232 6331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:35949 every 8 connection(s)
I20260924 23:16:16.977589 6265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:16.978559 6332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:16.980965 6332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897: Bootstrap starting.
I20260924 23:16:16.981546 6332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:16.981797 6332 log.cc:824] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:16.982506 6332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897: No bootstrap required, opened a new log
I20260924 23:16:16.983877 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 6265
I20260924 23:16:16.983968 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:16.984120 1043 external_mini_cluster.cc:1428] Setting key 86ceeb6b48192847bb5eb77b1338f7c5
I20260924 23:16:16.984161 6332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f885c67d2b04627a2522a87d6c73897" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35949 } }
I20260924 23:16:16.984339 6332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:16.984375 6332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f885c67d2b04627a2522a87d6c73897, State: Initialized, Role: FOLLOWER
I20260924 23:16:16.984526 6332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f885c67d2b04627a2522a87d6c73897" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35949 } }
I20260924 23:16:16.984607 6332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:16.984638 6332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:16.984678 6332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:16.985190 6332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f885c67d2b04627a2522a87d6c73897" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35949 } }
I20260924 23:16:16.985292 6332 leader_election.cc:302] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f885c67d2b04627a2522a87d6c73897; no voters:
I20260924 23:16:16.985451 6332 leader_election.cc:288] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:16.985495 6337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:16.985623 6337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [term 1 LEADER]: Becoming Leader. State: Replica: 6f885c67d2b04627a2522a87d6c73897, State: Running, Role: LEADER
I20260924 23:16:16.985706 6332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:16.985711 6337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f885c67d2b04627a2522a87d6c73897" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35949 } }
I20260924 23:16:16.986055 6337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f885c67d2b04627a2522a87d6c73897. Latest consensus state: current_term: 1 leader_uuid: "6f885c67d2b04627a2522a87d6c73897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f885c67d2b04627a2522a87d6c73897" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35949 } } }
I20260924 23:16:16.986117 6337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:16.986306 6342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:16.986774 6342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:16.987246 6338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f885c67d2b04627a2522a87d6c73897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f885c67d2b04627a2522a87d6c73897" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 35949 } } }
I20260924 23:16:16.987322 6338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:16.988425 6342 catalog_manager.cc:1347] Generated new cluster ID: 2670571272ad4adb881835a468deeea9
I20260924 23:16:16.988479 6342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:16:16.990118 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:16.984648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55900 (local address 127.1.4.254:35949)
0924 23:16:16.984844 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:16.984853 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:16.984884 (+ 31us) server_negotiation.cc:407] Connection header received
0924 23:16:16.985003 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:16.985011 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:16.985081 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:16.985199 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:16.985432 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:16.987188 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:16.988285 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:16.988773 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:16.988888 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:16.989495 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:16.989498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:16.989499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:16.989666 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:16.989702 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:16.989712 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:16.989768 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:16.989842 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 23:16:16.989918 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260924 23:16:16.990923 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:16:17.011837 6342 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:17.012588 6342 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:17.018996 6342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f885c67d2b04627a2522a87d6c73897: Generated new TSK 0
I20260924 23:16:17.019219 6342 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 23:16:17.038379 6362 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 23:16:17.038480 6362 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 23:16:17.038817 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:17.023841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.1.4.254:35949)
0924 23:16:17.023892 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:17.023895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:17.023922 (+ 27us) server_negotiation.cc:407] Connection header received
0924 23:16:17.024018 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:17.024019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:17.024031 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:17.024099 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0924 23:16:17.024219 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:17.025006 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:17.025844 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:17.026175 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:17.026241 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:17.026408 (+ 167us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 23:16:17.038573 (+ 12165us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 23:16:17.038742 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55916: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":27}
W20260924 23:16:17.038894 6336 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55916: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 23:16:17.038944 6361 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:17.023711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:35949 (local address 127.0.0.1:55916)
0924 23:16:17.023860 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:17.023871 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:17.023937 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:17.024105 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:17.024106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:17.024107 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0924 23:16:17.024157 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:17.024159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:17.025095 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:17.025096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:17.025752 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:17.025756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:17.026249 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:17.026253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:17.026297 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:17.026302 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 23:16:17.038738 (+ 12436us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 23:16:17.038845 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.4.254:35949: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260924 23:16:17.039220 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 6265
2026-09-24T23:16:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (224 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 23:16:17.076267 1043 mini_oidc.cc:152] Starting JWKS server
I20260924 23:16:17.076968 1043 webserver.cc:460] Webserver started at https://127.0.0.1:37123/ using document root <none> and password file <none>
I20260924 23:16:17.077023 1043 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 23:16:17.077171 1043 webserver.cc:460] Webserver started at http://127.0.0.1:44269/ using document root <none> and password file <none>
2026-09-24T23:16:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:17Z Disabled control of system clock
I20260924 23:16:17.084476 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38253
--webserver_interface=127.1.4.254
--webserver_port=0
--builtin_ntp_servers=127.1.4.212:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:38253
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37123/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 23:16:17.162899 6368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:17.163112 6368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:17.163142 6368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:17.164340 6368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:17.164387 6368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:17.164403 6368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:17.164419 6368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:17.164434 6368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 23:16:17.164449 6368 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 23:16:17.164466 6368 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37123/jwks.json
I20260924 23:16:17.166218 6368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:38253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37123/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:17.166495 6368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:17.166788 6368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:16:17.171602 6368 server_base.cc:1034] running on GCE node
W20260924 23:16:17.171577 6376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:17.171595 6374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:17.171574 6373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:17.171928 6368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:17.172210 6368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:17.173364 6368 hybrid_clock.cc:648] HybridClock initialized: now 1790291777173346 us; error 27 us; skew 500 ppm
I20260924 23:16:17.174624 6368 webserver.cc:460] Webserver started at http://127.1.4.254:40535/ using document root <none> and password file <none>
I20260924 23:16:17.174911 6368 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:17.174966 6368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:17.175137 6368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:17.176235 6368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "acf7325bc1234772b3c733a0da271883"
format_stamp: "Formatted at 2026-09-24 23:16:17 on dist-test-slave-s1hw"
server_key: "69bfb7369021359f4297f2a0a62ccea0"
server_key_iv: "9ff80b134cb6730af07438b97a409f29"
server_key_version: "encryptionkey@0"
I20260924 23:16:17.176564 6368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "acf7325bc1234772b3c733a0da271883"
format_stamp: "Formatted at 2026-09-24 23:16:17 on dist-test-slave-s1hw"
server_key: "69bfb7369021359f4297f2a0a62ccea0"
server_key_iv: "9ff80b134cb6730af07438b97a409f29"
server_key_version: "encryptionkey@0"
I20260924 23:16:17.177802 6368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:16:17.178535 6382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:17.178709 6368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:16:17.178790 6368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "acf7325bc1234772b3c733a0da271883"
format_stamp: "Formatted at 2026-09-24 23:16:17 on dist-test-slave-s1hw"
server_key: "69bfb7369021359f4297f2a0a62ccea0"
server_key_iv: "9ff80b134cb6730af07438b97a409f29"
server_key_version: "encryptionkey@0"
I20260924 23:16:17.178858 6368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:17.197482 6368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:17.197793 6368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:17.197923 6368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:17.201501 6368 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:38253
I20260924 23:16:17.201558 6434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:38253 every 8 connection(s)
I20260924 23:16:17.201875 6368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:17.202530 6435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:17.205154 6435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883: Bootstrap starting.
I20260924 23:16:17.205711 6435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:17.205966 6435 log.cc:824] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:17.206624 6435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883: No bootstrap required, opened a new log
I20260924 23:16:17.208266 6435 raft_consensus.cc:348] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf7325bc1234772b3c733a0da271883" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38253 } }
I20260924 23:16:17.208410 6435 raft_consensus.cc:374] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:17.208453 6435 raft_consensus.cc:729] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acf7325bc1234772b3c733a0da271883, State: Initialized, Role: FOLLOWER
I20260924 23:16:17.208597 6435 consensus_queue.cc:260] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf7325bc1234772b3c733a0da271883" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38253 } }
I20260924 23:16:17.208628 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 6368
I20260924 23:16:17.208678 6435 raft_consensus.cc:388] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:17.208703 6435 raft_consensus.cc:482] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:17.208738 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:17.208750 6435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:17.208912 1043 external_mini_cluster.cc:1428] Setting key 43959d1cba0b1fb568bdd88a8c06e48a
I20260924 23:16:17.209375 6435 raft_consensus.cc:504] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf7325bc1234772b3c733a0da271883" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38253 } }
I20260924 23:16:17.209484 6435 leader_election.cc:302] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acf7325bc1234772b3c733a0da271883; no voters:
I20260924 23:16:17.209645 6435 leader_election.cc:288] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:17.209672 6440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:17.209795 6440 raft_consensus.cc:686] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [term 1 LEADER]: Becoming Leader. State: Replica: acf7325bc1234772b3c733a0da271883, State: Running, Role: LEADER
I20260924 23:16:17.209887 6435 sys_catalog.cc:564] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:17.209883 6440 consensus_queue.cc:237] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf7325bc1234772b3c733a0da271883" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38253 } }
I20260924 23:16:17.210475 6441 sys_catalog.cc:455] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acf7325bc1234772b3c733a0da271883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf7325bc1234772b3c733a0da271883" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38253 } } }
I20260924 23:16:17.210558 6441 sys_catalog.cc:458] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:17.210711 6442 sys_catalog.cc:455] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acf7325bc1234772b3c733a0da271883. Latest consensus state: current_term: 1 leader_uuid: "acf7325bc1234772b3c733a0da271883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acf7325bc1234772b3c733a0da271883" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38253 } } }
I20260924 23:16:17.210767 6442 sys_catalog.cc:458] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:17.211120 6449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:17.211651 6449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:17.213189 6449 catalog_manager.cc:1347] Generated new cluster ID: 469a597b9b7445d683dcc5723eff9cf3
I20260924 23:16:17.213241 6449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:16:17.214751 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:17.209372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.1.4.254:38253)
0924 23:16:17.209650 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:17.209660 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:17.209694 (+ 34us) server_negotiation.cc:407] Connection header received
0924 23:16:17.209815 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:17.209822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:17.209893 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:17.210260 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:17.210355 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:17.211790 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:17.212693 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:17.213261 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:17.213414 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:17.214077 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:17.214081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:17.214081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:17.214278 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:17.214320 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:17.214335 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:17.214402 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:17.214456 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 23:16:17.214543 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":135,"threads_started":1}
I20260924 23:16:17.215579 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:16:17.227463 6449 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:17.228056 6449 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:17.231395 6449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acf7325bc1234772b3c733a0da271883: Generated new TSK 0
I20260924 23:16:17.231588 6449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:16:17.249014 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:17.245301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52184 (local address 127.1.4.254:38253)
0924 23:16:17.245357 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:17.245359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:17.245425 (+ 66us) server_negotiation.cc:407] Connection header received
0924 23:16:17.245513 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:17.245514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:17.245537 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:17.245617 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0924 23:16:17.245704 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:17.246484 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:17.247239 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:17.247576 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:17.247634 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:17.247826 (+ 192us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 23:16:17.248805 (+ 979us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37123/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 23:16:17.248938 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52184: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37123/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 23:16:17.249080 6439 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52184: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37123/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 23:16:17.250945 6464 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:17.245215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38253 (local address 127.0.0.1:52184)
0924 23:16:17.245369 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:17.245378 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:17.245450 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:17.245615 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:17.245616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:17.245618 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 23:16:17.245665 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:17.245667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:17.246567 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:17.246567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:17.247170 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:17.247172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:17.247663 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:17.247666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:17.247718 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:17.247724 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 23:16:17.248917 (+ 1193us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37123/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 23:16:17.250840 (+ 1923us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.4.254:38253: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37123/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":1689,"thread_start_us":86,"threads_started":1}
I20260924 23:16:17.251257 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 6368
E20260924 23:16:17.260125 6465 webserver.cc:533] Webserver: sslize failed:
E20260924 23:16:17.260207 6465 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T23:16:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (211 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T23:16:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:17Z Disabled control of system clock
I20260924 23:16:17.294636 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:41731
--webserver_interface=127.1.4.254
--webserver_port=0
--builtin_ntp_servers=127.1.4.212:35657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:41731
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/testchainca.cert with env {}
W20260924 23:16:17.374122 6469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:17.374344 6469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:17.374366 6469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:17.375569 6469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:17.375612 6469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:17.375623 6469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:17.375635 6469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:17.375646 6469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 23:16:17.375658 6469 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 23:16:17.375676 6469 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 23:16:17.377188 6469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:35657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:41731
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:41731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:17.377393 6469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:17.377565 6469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:17.382380 6474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:17.382387 6475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:17.382387 6477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:17.382709 6469 server_base.cc:1034] running on GCE node
I20260924 23:16:17.382948 6469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:17.383247 6469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:17.384441 6469 hybrid_clock.cc:648] HybridClock initialized: now 1790291777384420 us; error 31 us; skew 500 ppm
I20260924 23:16:17.385840 6469 webserver.cc:460] Webserver started at http://127.1.4.254:41865/ using document root <none> and password file <none>
I20260924 23:16:17.386086 6469 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:17.386139 6469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:17.386266 6469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:17.387351 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "b8e5a0bae3e0482b90fbd2dcd0b08c87"
format_stamp: "Formatted at 2026-09-24 23:16:17 on dist-test-slave-s1hw"
server_key: "358c251b84a9a6bba3f6f0da478b24d2"
server_key_iv: "5492440c59858f364af45e8f324d04bd"
server_key_version: "encryptionkey@0"
I20260924 23:16:17.387754 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "b8e5a0bae3e0482b90fbd2dcd0b08c87"
format_stamp: "Formatted at 2026-09-24 23:16:17 on dist-test-slave-s1hw"
server_key: "358c251b84a9a6bba3f6f0da478b24d2"
server_key_iv: "5492440c59858f364af45e8f324d04bd"
server_key_version: "encryptionkey@0"
I20260924 23:16:17.389151 6469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:16:17.389909 6483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:17.390110 6469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:16:17.390197 6469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "b8e5a0bae3e0482b90fbd2dcd0b08c87"
format_stamp: "Formatted at 2026-09-24 23:16:17 on dist-test-slave-s1hw"
server_key: "358c251b84a9a6bba3f6f0da478b24d2"
server_key_iv: "5492440c59858f364af45e8f324d04bd"
server_key_version: "encryptionkey@0"
I20260924 23:16:17.390260 6469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:17.402859 6469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:17.403184 6469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:17.403316 6469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:17.407095 6469 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:41731
I20260924 23:16:17.407135 6535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:41731 every 8 connection(s)
I20260924 23:16:17.407528 6469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:17.408475 6536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:17.408751 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 6469
I20260924 23:16:17.408865 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:17.409173 1043 external_mini_cluster.cc:1428] Setting key 1fa60f31ae838c9189dcdaf06da10ef8
I20260924 23:16:17.411109 6536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87: Bootstrap starting.
I20260924 23:16:17.411752 6536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:17.412088 6536 log.cc:824] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:17.412834 6536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87: No bootstrap required, opened a new log
I20260924 23:16:17.414548 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:17.409677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.1.4.254:41731)
0924 23:16:17.409851 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:17.409860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:17.409888 (+ 28us) server_negotiation.cc:407] Connection header received
0924 23:16:17.410293 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:17.410299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:17.410361 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:17.410510 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:17.410758 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:17.411695 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:17.412690 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:17.413142 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:17.413242 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:17.413886 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:17.413890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:17.413890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:17.414068 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:17.414102 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:17.414113 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:17.414164 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:17.414262 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 23:16:17.414328 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260924 23:16:17.414935 6536 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e5a0bae3e0482b90fbd2dcd0b08c87" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41731 } }
I20260924 23:16:17.415088 6536 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:17.415153 6536 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e5a0bae3e0482b90fbd2dcd0b08c87, State: Initialized, Role: FOLLOWER
I20260924 23:16:17.415268 6536 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e5a0bae3e0482b90fbd2dcd0b08c87" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41731 } }
I20260924 23:16:17.415347 6536 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:17.415406 6536 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:17.415450 6536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:17.415472 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:16:17.415995 6536 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e5a0bae3e0482b90fbd2dcd0b08c87" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41731 } }
I20260924 23:16:17.416092 6536 leader_election.cc:302] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8e5a0bae3e0482b90fbd2dcd0b08c87; no voters:
I20260924 23:16:17.416245 6536 leader_election.cc:288] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:17.416307 6541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:17.416463 6541 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [term 1 LEADER]: Becoming Leader. State: Replica: b8e5a0bae3e0482b90fbd2dcd0b08c87, State: Running, Role: LEADER
I20260924 23:16:17.416493 6536 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:17.416545 6541 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e5a0bae3e0482b90fbd2dcd0b08c87" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41731 } }
I20260924 23:16:17.416926 6542 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8e5a0bae3e0482b90fbd2dcd0b08c87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e5a0bae3e0482b90fbd2dcd0b08c87" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41731 } } }
I20260924 23:16:17.417028 6542 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:17.416913 6543 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8e5a0bae3e0482b90fbd2dcd0b08c87. Latest consensus state: current_term: 1 leader_uuid: "b8e5a0bae3e0482b90fbd2dcd0b08c87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e5a0bae3e0482b90fbd2dcd0b08c87" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41731 } } }
I20260924 23:16:17.417212 6543 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:17.417289 6549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:17.417874 6549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:17.419350 6549 catalog_manager.cc:1347] Generated new cluster ID: 741f6283510d48dfa00ab4fe7979b265
I20260924 23:16:17.419423 6549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:16:17.431444 6549 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:17.432255 6549 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:17.436237 6549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8e5a0bae3e0482b90fbd2dcd0b08c87: Generated new TSK 0
I20260924 23:16:17.436437 6549 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:16:17.500108 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:17.442152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51146 (local address 127.1.4.254:41731)
0924 23:16:17.442195 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:17.442197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:17.442302 (+ 105us) server_negotiation.cc:407] Connection header received
0924 23:16:17.442355 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:17.442356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:17.442367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:17.442426 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0924 23:16:17.442556 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:17.443030 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:17.444222 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:17.444418 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:17.444482 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:17.444682 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 23:16:17.499877 (+ 55195us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 23:16:17.500044 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51146: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 23:16:17.500175 6539 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51146: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 23:16:17.500285 6565 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:17.442056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:41731 (local address 127.0.0.1:51146)
0924 23:16:17.442212 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:17.442223 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:17.442291 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:17.442435 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:17.442436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:17.442438 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 23:16:17.442498 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:17.442500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:17.443135 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:17.443136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:17.444115 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:17.444119 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:17.444490 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:17.444494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:17.444555 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:17.444561 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 23:16:17.500035 (+ 55474us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 23:16:17.500183 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.4.254:41731: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260924 23:16:17.500597 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 6469
2026-09-24T23:16:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (229 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:16:17 dist-test-slave-s1hw krb5kdc[6577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:16:17 dist-test-slave-s1hw krb5kdc[6577](info): set up 2 sockets
Sep 24 23:16:17 dist-test-slave-s1hw krb5kdc[6577](info): commencing operation
krb5kdc: starting...
W20260924 23:16:18.623557 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:16:18 dist-test-slave-s1hw krb5kdc[6577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:16:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:16:18.735994 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:36099
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:36099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:18.814652 6590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:18.814885 6590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:18.814917 6590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:18.816141 6590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:18.816186 6590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:18.816203 6590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:18.816219 6590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:18.816234 6590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:18.817698 6590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:36099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:36099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:18.817946 6590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:18.818147 6590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:18.823019 6596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:18.822996 6598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:18.823019 6595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:18.823164 6590 server_base.cc:1034] running on GCE node
I20260924 23:16:18.823498 6590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:18.823760 6590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:18.824913 6590 hybrid_clock.cc:648] HybridClock initialized: now 1790291778824880 us; error 44 us; skew 500 ppm
Sep 24 23:16:18 dist-test-slave-s1hw krb5kdc[6577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291778, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:18.831043 6590 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:16:18.831573 6590 webserver.cc:460] Webserver started at http://127.1.4.254:38119/ using document root <none> and password file <none>
I20260924 23:16:18.831792 6590 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:18.831841 6590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:18.831974 6590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:18.833017 6590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "81292023ce734a3591368c9e47648737"
format_stamp: "Formatted at 2026-09-24 23:16:18 on dist-test-slave-s1hw"
server_key: "1c716381cdcf3f93533883d8faa0aa7d"
server_key_iv: "ec3bbeb0cb7bb9ac5d9e9e0e6b985c20"
server_key_version: "encryptionkey@0"
I20260924 23:16:18.833344 6590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "81292023ce734a3591368c9e47648737"
format_stamp: "Formatted at 2026-09-24 23:16:18 on dist-test-slave-s1hw"
server_key: "1c716381cdcf3f93533883d8faa0aa7d"
server_key_iv: "ec3bbeb0cb7bb9ac5d9e9e0e6b985c20"
server_key_version: "encryptionkey@0"
I20260924 23:16:18.834609 6590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:18.835323 6605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:18.835511 6590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:18.835599 6590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "81292023ce734a3591368c9e47648737"
format_stamp: "Formatted at 2026-09-24 23:16:18 on dist-test-slave-s1hw"
server_key: "1c716381cdcf3f93533883d8faa0aa7d"
server_key_iv: "ec3bbeb0cb7bb9ac5d9e9e0e6b985c20"
server_key_version: "encryptionkey@0"
I20260924 23:16:18.835662 6590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:18.875140 6590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:18.877493 6590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:18.877630 6590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:18.882165 6590 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:36099
I20260924 23:16:18.882220 6657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:36099 every 8 connection(s)
I20260924 23:16:18.882607 6590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:18.883708 6658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:18.886997 6658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737: Bootstrap starting.
I20260924 23:16:18.887755 6658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:18.888087 6658 log.cc:824] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:18.888880 6658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737: No bootstrap required, opened a new log
I20260924 23:16:18.890764 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 6590
I20260924 23:16:18.890865 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:18.891062 1043 external_mini_cluster.cc:1428] Setting key 365b49abe7e515b97912a9f2d08a8057
I20260924 23:16:18.891234 6658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81292023ce734a3591368c9e47648737" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36099 } }
I20260924 23:16:18.891453 6658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:18.891500 6658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81292023ce734a3591368c9e47648737, State: Initialized, Role: FOLLOWER
I20260924 23:16:18.891633 6658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81292023ce734a3591368c9e47648737" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36099 } }
I20260924 23:16:18.891711 6658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:18.891741 6658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:18.891780 6658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:18.892890 6658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81292023ce734a3591368c9e47648737" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36099 } }
I20260924 23:16:18.893007 6658 leader_election.cc:302] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81292023ce734a3591368c9e47648737; no voters:
I20260924 23:16:18.893172 6658 leader_election.cc:288] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:18.893251 6663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:18.893417 6663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [term 1 LEADER]: Becoming Leader. State: Replica: 81292023ce734a3591368c9e47648737, State: Running, Role: LEADER
I20260924 23:16:18.893443 6658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:18.893627 6663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81292023ce734a3591368c9e47648737" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36099 } }
I20260924 23:16:18.894033 6663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81292023ce734a3591368c9e47648737. Latest consensus state: current_term: 1 leader_uuid: "81292023ce734a3591368c9e47648737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81292023ce734a3591368c9e47648737" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36099 } } }
I20260924 23:16:18.894064 6664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81292023ce734a3591368c9e47648737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81292023ce734a3591368c9e47648737" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 36099 } } }
W20260924 23:16:18.895190 6677 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 23:16:18.894160 6663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:18.894179 6664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737 [sys.catalog]: This master's current role is: LEADER
W20260924 23:16:18.896078 6677 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:16:18.896291 6679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:18.896502 6679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:18.898145 6679 catalog_manager.cc:1347] Generated new cluster ID: 17f1a652783a455198d2f12613c5bd62
I20260924 23:16:18.898190 6679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:16:18 dist-test-slave-s1hw krb5kdc[6577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:18.906134 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:18.891613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.1.4.254:36099)
0924 23:16:18.891793 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:18.891804 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:18.891836 (+ 32us) server_negotiation.cc:407] Connection header received
0924 23:16:18.891976 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:18.891984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:18.892040 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:18.892189 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:18.892734 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:18.894139 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:18.895153 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:18.895652 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:18.895750 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:18.901724 (+ 5974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:18.901729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:18.901730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:18.901744 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:18.903942 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:18.904471 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:18.904475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:18.904475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:18.904589 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:18.904790 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:18.904790 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:18.904791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:18.905133 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:18.905295 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:18.905757 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 23:16:18.905912 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260924 23:16:18.914247 6679 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:18.915079 6679 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:18.924587 6679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81292023ce734a3591368c9e47648737: Generated new TSK 0
I20260924 23:16:18.924826 6679 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:16:18.952572 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:36099
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:19.034256 6684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:19.034468 6684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:19.034502 6684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:19.035672 6684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:19.035712 6684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:19.035744 6684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:16:19.037209 6684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:36099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:19.037448 6684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:19.037655 6684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:19.042524 6690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:19.042528 6692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:19.042580 6689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:19.042694 6684 server_base.cc:1034] running on GCE node
I20260924 23:16:19.042951 6684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:19.043167 6684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:19.044312 6684 hybrid_clock.cc:648] HybridClock initialized: now 1790291779044280 us; error 54 us; skew 500 ppm
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[6577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291779, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:19.046862 6684 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:16:19.047297 6684 webserver.cc:460] Webserver started at http://127.1.4.193:39173/ using document root <none> and password file <none>
I20260924 23:16:19.047535 6684 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:19.047580 6684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:19.047694 6684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:19.048734 6684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "39015e15511049dfb489b6a105e678b3"
format_stamp: "Formatted at 2026-09-24 23:16:19 on dist-test-slave-s1hw"
server_key: "ec2928400712d34c92c5b896269fe579"
server_key_iv: "cb09b23ece341924de375b8ba61ceb99"
server_key_version: "encryptionkey@0"
I20260924 23:16:19.049057 6684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "39015e15511049dfb489b6a105e678b3"
format_stamp: "Formatted at 2026-09-24 23:16:19 on dist-test-slave-s1hw"
server_key: "ec2928400712d34c92c5b896269fe579"
server_key_iv: "cb09b23ece341924de375b8ba61ceb99"
server_key_version: "encryptionkey@0"
I20260924 23:16:19.050246 6684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:19.050936 6699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:19.051076 6684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:19.051149 6684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "39015e15511049dfb489b6a105e678b3"
format_stamp: "Formatted at 2026-09-24 23:16:19 on dist-test-slave-s1hw"
server_key: "ec2928400712d34c92c5b896269fe579"
server_key_iv: "cb09b23ece341924de375b8ba61ceb99"
server_key_version: "encryptionkey@0"
I20260924 23:16:19.051214 6684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:19.061714 6684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:19.063531 6684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:19.063666 6684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:19.063866 6684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:19.064150 6684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:19.064181 6684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:19.064203 6684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:19.064225 6684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:19.069607 6684 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:32935
I20260924 23:16:19.069665 6812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:32935 every 8 connection(s)
I20260924 23:16:19.069962 6684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[6577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291779, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:19.078285 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 6684
I20260924 23:16:19.078390 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:16:19.078629 1043 external_mini_cluster.cc:1428] Setting key c603026a2d38f966b8ef92bc0cb5cf53
I20260924 23:16:19.079129 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:19.070657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:43967 (local address 127.1.4.254:36099)
0924 23:16:19.070716 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:19.070718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:19.071088 (+ 370us) server_negotiation.cc:407] Connection header received
0924 23:16:19.072070 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:19.072071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:19.072081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:19.072134 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:19.072773 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.073283 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.074237 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.074430 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.074479 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:19.076422 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:19.076423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:19.076424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:19.076431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:19.077719 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:19.078103 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:19.078104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:19.078104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:19.078171 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:19.078417 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:19.078417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:19.078418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:19.078568 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:19.078626 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:19.078940 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 23:16:19.079040 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:16:19.079298 6815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:19.070679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:36099 (local address 127.1.4.193:43967)
0924 23:16:19.070911 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:19.070973 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:19.071961 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:19.072206 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:19.072209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:19.072495 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:19.072691 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:19.072695 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.073346 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.073347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:19.074158 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:19.074160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.074490 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.074491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:19.074588 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:19.075016 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:19.075024 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:19.076342 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:19.077804 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:19.077810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:19.077812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:19.078031 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:19.078245 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:19.078246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:19.078246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:19.078359 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:19.078652 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:19.078659 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:19.078810 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 23:16:19.078905 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 23:16:19.079077 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":121,"threads_started":1}
I20260924 23:16:19.079836 6813 heartbeater.cc:347] Connected to a master server at 127.1.4.254:36099
I20260924 23:16:19.079928 6813 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:19.080104 6813 heartbeater.cc:511] Master 127.1.4.254:36099 requested a full tablet report, sending...
I20260924 23:16:19.080554 6622 ts_manager.cc:194] Registered new tserver with Master: 39015e15511049dfb489b6a105e678b3 (127.1.4.193:32935)
I20260924 23:16:19.081148 6622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:43967
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:16:19.102319 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:36099
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:19.182155 6818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:19.182363 6818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:19.182384 6818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:19.183525 6818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:19.183562 6818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:19.183588 6818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:16:19.185014 6818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:36099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:19.185210 6818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:19.185405 6818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:19.190464 6823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:19.190703 6818 server_base.cc:1034] running on GCE node
W20260924 23:16:19.190480 6826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:19.190477 6824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:19.191128 6818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:19.191430 6818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:19.192608 6818 hybrid_clock.cc:648] HybridClock initialized: now 1790291779192589 us; error 29 us; skew 500 ppm
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[6577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291779, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:19.195387 6818 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:16:19.195818 6818 webserver.cc:460] Webserver started at http://127.1.4.194:36371/ using document root <none> and password file <none>
I20260924 23:16:19.195992 6818 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:19.196029 6818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:19.196122 6818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:19.197104 6818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "0626a9758aae49dcacec343292394d61"
format_stamp: "Formatted at 2026-09-24 23:16:19 on dist-test-slave-s1hw"
server_key: "7c61cf4494ae09dd7b0090aea33208f5"
server_key_iv: "0d654236da746d46e4e816691950e4fe"
server_key_version: "encryptionkey@0"
I20260924 23:16:19.197389 6818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "0626a9758aae49dcacec343292394d61"
format_stamp: "Formatted at 2026-09-24 23:16:19 on dist-test-slave-s1hw"
server_key: "7c61cf4494ae09dd7b0090aea33208f5"
server_key_iv: "0d654236da746d46e4e816691950e4fe"
server_key_version: "encryptionkey@0"
I20260924 23:16:19.198585 6818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:19.199221 6833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:19.199388 6818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:19.199493 6818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "0626a9758aae49dcacec343292394d61"
format_stamp: "Formatted at 2026-09-24 23:16:19 on dist-test-slave-s1hw"
server_key: "7c61cf4494ae09dd7b0090aea33208f5"
server_key_iv: "0d654236da746d46e4e816691950e4fe"
server_key_version: "encryptionkey@0"
I20260924 23:16:19.199550 6818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:19.218611 6818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:19.220634 6818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:19.220747 6818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:19.220938 6818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:19.221258 6818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:19.221290 6818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:19.221312 6818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:19.221325 6818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:19.226521 6818 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:35743
I20260924 23:16:19.226562 6946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:35743 every 8 connection(s)
I20260924 23:16:19.226884 6818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[6577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291779, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:19.235612 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:19.227631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:35215 (local address 127.1.4.254:36099)
0924 23:16:19.227696 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:19.227698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:19.227856 (+ 158us) server_negotiation.cc:407] Connection header received
0924 23:16:19.228808 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:19.228809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:19.228820 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:19.228878 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:19.229545 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.230042 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.231024 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.231235 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.231292 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:19.233341 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:19.233342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:19.233343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:19.233350 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:19.234405 (+ 1055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:19.234752 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:19.234752 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:19.234753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:19.234796 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:19.235028 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:19.235029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:19.235029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:19.235134 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:19.235182 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:19.235471 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 23:16:19.235547 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 23:16:19.235710 6949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:19.227585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:36099 (local address 127.1.4.194:35215)
0924 23:16:19.227739 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:19.227775 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:19.228695 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:19.228954 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:19.228958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:19.229228 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:19.229451 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:19.229456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.230137 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.230138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:19.230939 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:19.230941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.231306 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.231307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:19.231448 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:19.231916 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:19.231924 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:19.233257 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:19.234481 (+ 1224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:19.234486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:19.234488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:19.234698 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:19.234844 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:19.234844 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:19.234844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:19.234936 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:19.235179 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:19.235182 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:19.235301 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 23:16:19.235410 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 23:16:19.235535 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260924 23:16:19.236169 6947 heartbeater.cc:347] Connected to a master server at 127.1.4.254:36099
I20260924 23:16:19.236256 6947 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:19.236418 6947 heartbeater.cc:511] Master 127.1.4.254:36099 requested a full tablet report, sending...
I20260924 23:16:19.236763 6622 ts_manager.cc:194] Registered new tserver with Master: 0626a9758aae49dcacec343292394d61 (127.1.4.194:35743)
I20260924 23:16:19.237040 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 6818
I20260924 23:16:19.237128 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:16:19.237152 6622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:35215
I20260924 23:16:19.237314 1043 external_mini_cluster.cc:1428] Setting key 564be56ebe8423f7512aba84891822df
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:16:19.259763 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:36099
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:19.338801 6952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:19.339025 6952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:19.339056 6952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:19.340238 6952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:19.340281 6952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:19.340315 6952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:16:19.341742 6952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:36099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:19.341976 6952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:19.342187 6952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:19.347023 6960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:19.347023 6958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:19.347002 6957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:19.347311 6952 server_base.cc:1034] running on GCE node
I20260924 23:16:19.347550 6952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:19.347785 6952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:19.348920 6952 hybrid_clock.cc:648] HybridClock initialized: now 1790291779348903 us; error 35 us; skew 500 ppm
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[6577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291779, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:19.351475 6952 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:16:19.351922 6952 webserver.cc:460] Webserver started at http://127.1.4.195:44435/ using document root <none> and password file <none>
I20260924 23:16:19.352109 6952 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:19.352149 6952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:19.352236 6952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:19.353240 6952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "7b5317caa62e4324a7dfe986c31ab156"
format_stamp: "Formatted at 2026-09-24 23:16:19 on dist-test-slave-s1hw"
server_key: "a6217500d82a5c13b18f9dabd8edb991"
server_key_iv: "e16176f79351f4ec6ac8812aa0ba6126"
server_key_version: "encryptionkey@0"
I20260924 23:16:19.353525 6952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "7b5317caa62e4324a7dfe986c31ab156"
format_stamp: "Formatted at 2026-09-24 23:16:19 on dist-test-slave-s1hw"
server_key: "a6217500d82a5c13b18f9dabd8edb991"
server_key_iv: "e16176f79351f4ec6ac8812aa0ba6126"
server_key_version: "encryptionkey@0"
I20260924 23:16:19.354692 6952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 23:16:19.355397 6967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:19.355552 6952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:19.355630 6952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "7b5317caa62e4324a7dfe986c31ab156"
format_stamp: "Formatted at 2026-09-24 23:16:19 on dist-test-slave-s1hw"
server_key: "a6217500d82a5c13b18f9dabd8edb991"
server_key_iv: "e16176f79351f4ec6ac8812aa0ba6126"
server_key_version: "encryptionkey@0"
I20260924 23:16:19.355690 6952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:19.376116 6952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:19.378322 6952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:19.378463 6952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:19.378700 6952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:19.379022 6952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:19.379061 6952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:19.379091 6952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:19.379114 6952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:19.384526 6952 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:45705
I20260924 23:16:19.384574 7080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:45705 every 8 connection(s)
I20260924 23:16:19.384868 6952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[6577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291779, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:19.393867 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:19.385599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:56703 (local address 127.1.4.254:36099)
0924 23:16:19.385645 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:19.385648 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:19.385833 (+ 185us) server_negotiation.cc:407] Connection header received
0924 23:16:19.386797 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:19.386798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:19.386811 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:19.386860 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:19.387530 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.388021 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.389068 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.389248 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.389301 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:19.391325 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:19.391327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:19.391327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:19.391335 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:19.392580 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:19.392991 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:19.392992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:19.392992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:19.393050 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:19.393257 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:19.393257 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:19.393258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:19.393365 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:19.393421 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:19.393699 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 23:16:19.393774 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:16:19.394049 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:19.385537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:36099 (local address 127.1.4.195:56703)
0924 23:16:19.385704 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:19.385745 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:19.386674 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:19.386943 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:19.386947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:19.387204 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:19.387450 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:19.387455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.388095 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.388096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:19.388994 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:19.388996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.389312 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.389312 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:19.389432 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:19.389886 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:19.389894 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:19.391229 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:19.392652 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:19.392658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:19.392661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:19.392924 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:19.393100 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:19.393100 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:19.393101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:19.393204 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:19.393418 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:19.393423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:19.393566 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 23:16:19.393679 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 23:16:19.393834 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260924 23:16:19.394523 7081 heartbeater.cc:347] Connected to a master server at 127.1.4.254:36099
I20260924 23:16:19.394614 7081 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:19.394603 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 6952
I20260924 23:16:19.394740 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:16:19.394810 7081 heartbeater.cc:511] Master 127.1.4.254:36099 requested a full tablet report, sending...
I20260924 23:16:19.395037 1043 external_mini_cluster.cc:1428] Setting key 8c0b5f2af20076399ba5b781f2c793bb
I20260924 23:16:19.395179 6622 ts_manager.cc:194] Registered new tserver with Master: 7b5317caa62e4324a7dfe986c31ab156 (127.1.4.195:45705)
I20260924 23:16:19.395624 6622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:56703
I20260924 23:16:19.395848 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/test-admin.keytab.
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[6577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:19.411842 1043 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 23:16:19.413558 7088 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:19.412716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.1.4.193:32935)
0924 23:16:19.412977 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:19.412982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:19.412994 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:16:19.413021 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:19.413023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:19.413044 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:19.413122 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:19.413477 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51118: BlockingRecv error: recv got EOF from 127.0.0.1:51118 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":97,"threads_started":1}
W20260924 23:16:19.413707 1043 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/krb5cc)
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[6577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:19.414746 1043 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[6577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:19.415798 1043 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[6577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.193@KRBTEST.COM
I20260924 23:16:19.423535 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:19.416452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51120 (local address 127.1.4.193:32935)
0924 23:16:19.416501 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:19.416503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:19.416549 (+ 46us) server_negotiation.cc:407] Connection header received
0924 23:16:19.416611 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:19.416612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:19.416622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:19.416667 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:19.417084 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.417821 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.418575 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.418873 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.418923 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:19.421106 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:19.421113 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:19.421113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:19.421120 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:19.422489 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:19.422801 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:19.422803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:19.422803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:19.422851 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:19.423042 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:19.423043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:19.423043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:19.423158 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:19.423256 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:19.423339 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 23:16:19.423443 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 23:16:19.425241 7088 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:19.424745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.1.4.193:32935)
0924 23:16:19.424792 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:19.424794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:19.424835 (+ 41us) server_negotiation.cc:407] Connection header received
0924 23:16:19.424893 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:19.424894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:19.424902 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:19.424937 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:19.425200 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51124: BlockingRecv error: recv got EOF from 127.0.0.1:51124 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 23:16:19.425364 1043 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/krb5cc' of type FILE
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[6577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:19.426263 1043 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[6577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:19.427054 1043 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[6577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.193@KRBTEST.COM
I20260924 23:16:19.434044 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:19.427623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.1.4.193:32935)
0924 23:16:19.427677 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:19.427678 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:19.427688 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:16:19.427738 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:19.427739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:19.427747 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:19.427788 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:19.428133 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.428607 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.429335 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.429544 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.429600 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:19.431710 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:19.431712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:19.431712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:19.431718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:19.433070 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:19.433356 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:19.433358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:19.433358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:19.433417 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:19.433579 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:19.433580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:19.433580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:19.433665 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:19.433729 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:19.433806 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 23:16:19.433952 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 23:16:19.435413 7088 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:19.434899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.1.4.193:32935)
0924 23:16:19.434943 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:19.434945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:19.435043 (+ 98us) server_negotiation.cc:407] Connection header received
0924 23:16:19.435086 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:19.435086 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:19.435094 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:19.435134 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:19.435343 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51140: BlockingRecv error: recv got EOF from 127.0.0.1:51140 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 23:16:19.435484 1043 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291755455739-1043-0/krb5kdc/krb5cc)
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[6577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:19.436396 1043 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[6577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:19.437391 1043 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[6577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.193@KRBTEST.COM
I20260924 23:16:19.444766 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:19.438020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51146 (local address 127.1.4.193:32935)
0924 23:16:19.438069 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:19.438070 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:19.438079 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:19.438135 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:19.438135 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:19.438143 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:19.438177 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:19.438515 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.439041 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.439848 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:19.440032 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:19.440094 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:19.442194 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:19.442195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:19.442196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:19.442202 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:19.443671 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:19.444009 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:19.444010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:19.444011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:19.444066 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:19.444305 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:19.444306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:19.444307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:19.444448 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:19.444544 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:19.444633 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 23:16:19.444701 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:16:19.445564 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 6684
I20260924 23:16:19.450019 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 6818
I20260924 23:16:19.453948 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 6952
I20260924 23:16:19.457592 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 6590
2026-09-24T23:16:19Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1784 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[7101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[7101](info): set up 2 sockets
Sep 24 23:16:19 dist-test-slave-s1hw krb5kdc[7101](info): commencing operation
krb5kdc: starting...
W20260924 23:16:20.415474 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:16:20 dist-test-slave-s1hw krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:16:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.254.keytab.
Entry for principal oryx/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.254.keytab.
I20260924 23:16:20.534423 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:42079
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.254.keytab
--principal=oryx/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:42079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:20.613055 7114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:20.613289 7114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:20.613323 7114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:20.614533 7114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:20.614591 7114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:20.614604 7114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:20.614615 7114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:20.614626 7114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:20.616166 7114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:42079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.254.keytab
--principal=oryx/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:42079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:20.616400 7114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:20.616626 7114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:16:20.621536 7114 server_base.cc:1034] running on GCE node
W20260924 23:16:20.621519 7119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:20.621519 7122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:20.621519 7120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:20.622051 7114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:20.622336 7114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:20.623441 7114 hybrid_clock.cc:648] HybridClock initialized: now 1790291780623418 us; error 33 us; skew 500 ppm
Sep 24 23:16:20 dist-test-slave-s1hw krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291780, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:20.625890 7114 init.cc:377] Logged in from keytab as oryx/127.1.4.254@KRBTEST.COM (short username oryx)
I20260924 23:16:20.626300 7114 webserver.cc:460] Webserver started at http://127.1.4.254:33553/ using document root <none> and password file <none>
I20260924 23:16:20.626525 7114 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:20.626574 7114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:20.626691 7114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:20.627800 7114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "c3a137b4a02d44709e97e6af2f7dd626"
format_stamp: "Formatted at 2026-09-24 23:16:20 on dist-test-slave-s1hw"
server_key: "215b62f01eae952b2d49da1dfc1cefa5"
server_key_iv: "f80d63be66c9f88bd79b3f20e424d3b8"
server_key_version: "encryptionkey@0"
I20260924 23:16:20.628113 7114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "c3a137b4a02d44709e97e6af2f7dd626"
format_stamp: "Formatted at 2026-09-24 23:16:20 on dist-test-slave-s1hw"
server_key: "215b62f01eae952b2d49da1dfc1cefa5"
server_key_iv: "f80d63be66c9f88bd79b3f20e424d3b8"
server_key_version: "encryptionkey@0"
I20260924 23:16:20.629318 7114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:16:20.629979 7129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:20.630122 7114 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:20.630201 7114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "c3a137b4a02d44709e97e6af2f7dd626"
format_stamp: "Formatted at 2026-09-24 23:16:20 on dist-test-slave-s1hw"
server_key: "215b62f01eae952b2d49da1dfc1cefa5"
server_key_iv: "f80d63be66c9f88bd79b3f20e424d3b8"
server_key_version: "encryptionkey@0"
I20260924 23:16:20.630255 7114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:20.661440 7114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:20.663621 7114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:20.663759 7114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:20.667305 7114 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:42079
I20260924 23:16:20.667342 7181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:42079 every 8 connection(s)
I20260924 23:16:20.667703 7114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:20.668677 7182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:20.668974 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 7114
I20260924 23:16:20.669070 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:20.669253 1043 external_mini_cluster.cc:1428] Setting key 0b7148da3484bf010763f037d636c58f
I20260924 23:16:20.671164 7182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626: Bootstrap starting.
I20260924 23:16:20.671722 7182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:20.671980 7182 log.cc:824] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:20.672571 7182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626: No bootstrap required, opened a new log
I20260924 23:16:20.674252 7182 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } }
I20260924 23:16:20.674381 7182 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:20.674412 7182 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a137b4a02d44709e97e6af2f7dd626, State: Initialized, Role: FOLLOWER
I20260924 23:16:20.674535 7182 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } }
I20260924 23:16:20.674613 7182 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:20.674643 7182 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:20.674687 7182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:20.675320 7182 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } }
I20260924 23:16:20.675451 7182 leader_election.cc:302] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3a137b4a02d44709e97e6af2f7dd626; no voters:
Sep 24 23:16:20 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM
I20260924 23:16:20.675604 7182 leader_election.cc:288] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:20.675647 7187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:20.675827 7187 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [term 1 LEADER]: Becoming Leader. State: Replica: c3a137b4a02d44709e97e6af2f7dd626, State: Running, Role: LEADER
I20260924 23:16:20.675844 7182 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:20.676065 7187 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } }
I20260924 23:16:20.676697 7189 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3a137b4a02d44709e97e6af2f7dd626. Latest consensus state: current_term: 1 leader_uuid: "c3a137b4a02d44709e97e6af2f7dd626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } }
I20260924 23:16:20.676771 7189 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:20.677292 7188 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3a137b4a02d44709e97e6af2f7dd626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } }
I20260924 23:16:20.677557 7188 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: This master's current role is: LEADER
W20260924 23:16:20.677634 7201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:16:20.677683 7201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:16:20.679507 7205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:20.679584 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:20.669787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45942 (local address 127.1.4.254:42079)
0924 23:16:20.669969 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:20.669978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:20.670004 (+ 26us) server_negotiation.cc:407] Connection header received
0924 23:16:20.670404 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:20.670410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:20.670461 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:20.670584 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:20.671186 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:20.672249 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:20.673016 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:20.673438 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:20.673533 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:20.676063 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:20.676067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:20.676067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:20.676079 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:20.678142 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:20.678485 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:20.678487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:20.678488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:20.678588 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:20.678745 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:20.678746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:20.678746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:20.679030 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:20.679159 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:20.679231 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 23:16:20.679358 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260924 23:16:20.679675 7205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:20.681066 7205 catalog_manager.cc:1347] Generated new cluster ID: 79e249b6f5f4479fa2bb7f5060558ed9
I20260924 23:16:20.681115 7205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:16:20.704934 7205 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:20.705469 7205 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:20.710278 7205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626: Generated new TSK 0
I20260924 23:16:20.710466 7205 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.193.keytab.
Entry for principal oryx/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.193.keytab.
I20260924 23:16:20.743670 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:42079
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.193.keytab
--principal=oryx/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:20.831118 7209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:20.831323 7209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:20.831346 7209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:20.832507 7209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:20.832551 7209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:20.832576 7209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:16:20.834031 7209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.193.keytab
--principal=oryx/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:42079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:20.834254 7209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:20.834466 7209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:20.839556 7215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:20.839545 7209 server_base.cc:1034] running on GCE node
W20260924 23:16:20.839530 7217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:20.839530 7214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:20.839947 7209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:20.840171 7209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:20.841320 7209 hybrid_clock.cc:648] HybridClock initialized: now 1790291780841300 us; error 37 us; skew 500 ppm
Sep 24 23:16:20 dist-test-slave-s1hw krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291780, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:20.848032 7209 init.cc:377] Logged in from keytab as oryx/127.1.4.193@KRBTEST.COM (short username oryx)
I20260924 23:16:20.848510 7209 webserver.cc:460] Webserver started at http://127.1.4.193:34479/ using document root <none> and password file <none>
I20260924 23:16:20.848723 7209 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:20.848771 7209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:20.848875 7209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:20.849920 7209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "08ab7c555b634abda9f23607f5ee397c"
format_stamp: "Formatted at 2026-09-24 23:16:20 on dist-test-slave-s1hw"
server_key: "e668dd492baeddca61abe92ddc836890"
server_key_iv: "4ea44b4df3a313bfdbf258b892216129"
server_key_version: "encryptionkey@0"
I20260924 23:16:20.850229 7209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "08ab7c555b634abda9f23607f5ee397c"
format_stamp: "Formatted at 2026-09-24 23:16:20 on dist-test-slave-s1hw"
server_key: "e668dd492baeddca61abe92ddc836890"
server_key_iv: "4ea44b4df3a313bfdbf258b892216129"
server_key_version: "encryptionkey@0"
I20260924 23:16:20.851611 7209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:20.852355 7224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:20.852515 7209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:20.852597 7209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "08ab7c555b634abda9f23607f5ee397c"
format_stamp: "Formatted at 2026-09-24 23:16:20 on dist-test-slave-s1hw"
server_key: "e668dd492baeddca61abe92ddc836890"
server_key_iv: "4ea44b4df3a313bfdbf258b892216129"
server_key_version: "encryptionkey@0"
I20260924 23:16:20.852670 7209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:20.859560 7209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:20.861747 7209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:20.861886 7209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:20.862093 7209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:20.862383 7209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:20.862418 7209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:20.862443 7209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:20.862470 7209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:20.868436 7209 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:34339
I20260924 23:16:20.868470 7337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:34339 every 8 connection(s)
I20260924 23:16:20.868801 7209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:16:20.868986 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 7209
I20260924 23:16:20.869102 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:16:20.869318 1043 external_mini_cluster.cc:1428] Setting key cc42f7630184f7e04b81c307f6a942ba
Sep 24 23:16:20 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291780, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.1.4.194@KRBTEST.COM; defaulting to no policy
I20260924 23:16:20.880096 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:20.869574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:42583 (local address 127.1.4.254:42079)
0924 23:16:20.869629 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:20.869631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:20.869864 (+ 233us) server_negotiation.cc:407] Connection header received
0924 23:16:20.870496 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:20.870498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:20.870511 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:20.870585 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:20.871945 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:20.872709 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:20.874199 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:20.874398 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:20.874486 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:20.877053 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:20.877055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:20.877055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:20.877063 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:20.878595 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:20.879083 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:20.879084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:20.879085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:20.879144 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:20.879415 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:20.879416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:20.879416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:20.879536 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:20.879604 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:20.879898 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 23:16:20.879994 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:16:20.880245 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:20.869548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.193:42583)
0924 23:16:20.869719 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:20.869763 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:20.870342 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:20.870761 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:20.870765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:20.871551 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:20.871833 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:20.871840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:20.872828 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:20.872829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:20.874109 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:20.874112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:20.874485 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:20.874487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:20.874629 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:20.875180 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:20.875191 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:20.876968 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:20.878672 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:20.878680 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:20.878682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:20.879003 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:20.879204 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:20.879205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:20.879205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:20.879320 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:20.879599 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:20.879614 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:20.879760 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 23:16:20.879857 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 23:16:20.880021 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.4.194@KRBTEST.COM" created.
I20260924 23:16:20.880806 7338 heartbeater.cc:347] Connected to a master server at 127.1.4.254:42079
I20260924 23:16:20.880915 7338 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:20.881111 7338 heartbeater.cc:511] Master 127.1.4.254:42079 requested a full tablet report, sending...
I20260924 23:16:20.881618 7146 ts_manager.cc:194] Registered new tserver with Master: 08ab7c555b634abda9f23607f5ee397c (127.1.4.193:34339)
I20260924 23:16:20.882462 7146 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:42583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.194.keytab.
Entry for principal oryx/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.194.keytab.
I20260924 23:16:20.894668 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:42079
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.194.keytab
--principal=oryx/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:20.975672 7343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:20.975951 7343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:20.975981 7343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:20.977296 7343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:20.977353 7343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:20.977382 7343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:16:20.978812 7343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.194.keytab
--principal=oryx/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:42079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:20.979027 7343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:20.979212 7343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:16:20.984182 7343 server_base.cc:1034] running on GCE node
W20260924 23:16:20.984195 7348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:20.984134 7351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:20.984134 7349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:20.984511 7343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:20.984756 7343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:20.985912 7343 hybrid_clock.cc:648] HybridClock initialized: now 1790291780985897 us; error 31 us; skew 500 ppm
Sep 24 23:16:20 dist-test-slave-s1hw krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291780, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:20.988644 7343 init.cc:377] Logged in from keytab as oryx/127.1.4.194@KRBTEST.COM (short username oryx)
I20260924 23:16:20.989137 7343 webserver.cc:460] Webserver started at http://127.1.4.194:34177/ using document root <none> and password file <none>
I20260924 23:16:20.989338 7343 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:20.989382 7343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:20.989501 7343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:20.990522 7343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "e85bb432f60642fb92fa4c620b6ac7a0"
format_stamp: "Formatted at 2026-09-24 23:16:20 on dist-test-slave-s1hw"
server_key: "a11046435b1c8ff1095b3ddb13148bd0"
server_key_iv: "75e4ca3b23e2642c567baaf4fe69d1c1"
server_key_version: "encryptionkey@0"
I20260924 23:16:20.990851 7343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "e85bb432f60642fb92fa4c620b6ac7a0"
format_stamp: "Formatted at 2026-09-24 23:16:20 on dist-test-slave-s1hw"
server_key: "a11046435b1c8ff1095b3ddb13148bd0"
server_key_iv: "75e4ca3b23e2642c567baaf4fe69d1c1"
server_key_version: "encryptionkey@0"
I20260924 23:16:20.992255 7343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:16:20.993077 7358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:20.993271 7343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:16:20.993361 7343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "e85bb432f60642fb92fa4c620b6ac7a0"
format_stamp: "Formatted at 2026-09-24 23:16:20 on dist-test-slave-s1hw"
server_key: "a11046435b1c8ff1095b3ddb13148bd0"
server_key_iv: "75e4ca3b23e2642c567baaf4fe69d1c1"
server_key_version: "encryptionkey@0"
I20260924 23:16:20.993438 7343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:21.019393 7343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:21.021423 7343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:21.021552 7343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:21.021798 7343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:21.022114 7343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:21.022147 7343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.022176 7343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:21.022193 7343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.027446 7343 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:37679
I20260924 23:16:21.027499 7471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:37679 every 8 connection(s)
I20260924 23:16:21.027801 7343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:16:21.029275 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 7343
I20260924 23:16:21.029378 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:16:21.029573 1043 external_mini_cluster.cc:1428] Setting key 8b3a6c697136a5db237117f1393ea1fa
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291780, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.1.4.195@KRBTEST.COM; defaulting to no policy
I20260924 23:16:21.038002 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.028593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:45455 (local address 127.1.4.254:42079)
0924 23:16:21.028670 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.028673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.028784 (+ 111us) server_negotiation.cc:407] Connection header received
0924 23:16:21.029415 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.029417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.029429 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.029494 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.030694 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.031203 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.032309 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.032586 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.032655 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.034913 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.034915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.034915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.034924 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.036713 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.037182 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.037183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.037184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.037247 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.037473 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.037474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.037474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.037593 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.037645 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.037870 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.037930 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 23:16:21.038257 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.028479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.194:45455)
0924 23:16:21.028655 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.028704 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.029270 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.029591 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.029595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.030306 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.030604 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.030612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.031286 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.031287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.032225 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.032227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.032672 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.032673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.032781 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.033363 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.033374 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.034819 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.036804 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.036810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.036812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.037105 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.037306 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.037306 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.037306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.037411 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.037641 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.037644 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.037766 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.037844 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.038021 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260924 23:16:21.038760 7472 heartbeater.cc:347] Connected to a master server at 127.1.4.254:42079
I20260924 23:16:21.038857 7472 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:21.039044 7472 heartbeater.cc:511] Master 127.1.4.254:42079 requested a full tablet report, sending...
I20260924 23:16:21.039435 7146 ts_manager.cc:194] Registered new tserver with Master: e85bb432f60642fb92fa4c620b6ac7a0 (127.1.4.194:37679)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.4.195@KRBTEST.COM" created.
I20260924 23:16:21.039839 7146 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:45455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.195.keytab.
Entry for principal oryx/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.195.keytab.
I20260924 23:16:21.052292 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:42079
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.195.keytab
--principal=oryx/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:21.132102 7477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:21.132319 7477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:21.132352 7477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:21.133572 7477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:21.133620 7477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:21.133656 7477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:16:21.135557 7477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.195.keytab
--principal=oryx/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:42079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:21.135824 7477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:21.136047 7477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:21.140941 7483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:21.140950 7485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:21.141047 7477 server_base.cc:1034] running on GCE node
W20260924 23:16:21.141166 7482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:21.141361 7477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:21.141594 7477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:21.142745 7477 hybrid_clock.cc:648] HybridClock initialized: now 1790291781142726 us; error 28 us; skew 500 ppm
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291781, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:21.145393 7477 init.cc:377] Logged in from keytab as oryx/127.1.4.195@KRBTEST.COM (short username oryx)
I20260924 23:16:21.145848 7477 webserver.cc:460] Webserver started at http://127.1.4.195:43247/ using document root <none> and password file <none>
I20260924 23:16:21.146055 7477 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:21.146111 7477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:21.146231 7477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:21.147243 7477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "3eec19b8e01544c7a91a6ab99e738b73"
format_stamp: "Formatted at 2026-09-24 23:16:21 on dist-test-slave-s1hw"
server_key: "f2ce0de9d24849dd794a3b040307ebc5"
server_key_iv: "548ffc99845afca4d328e7b35b3c5a36"
server_key_version: "encryptionkey@0"
I20260924 23:16:21.147596 7477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "3eec19b8e01544c7a91a6ab99e738b73"
format_stamp: "Formatted at 2026-09-24 23:16:21 on dist-test-slave-s1hw"
server_key: "f2ce0de9d24849dd794a3b040307ebc5"
server_key_iv: "548ffc99845afca4d328e7b35b3c5a36"
server_key_version: "encryptionkey@0"
I20260924 23:16:21.148856 7477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:21.149569 7492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.149726 7477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:21.149807 7477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "3eec19b8e01544c7a91a6ab99e738b73"
format_stamp: "Formatted at 2026-09-24 23:16:21 on dist-test-slave-s1hw"
server_key: "f2ce0de9d24849dd794a3b040307ebc5"
server_key_iv: "548ffc99845afca4d328e7b35b3c5a36"
server_key_version: "encryptionkey@0"
I20260924 23:16:21.149868 7477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:21.174290 7477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:21.176290 7477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:21.176437 7477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:21.176673 7477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:21.176983 7477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:21.177016 7477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.177044 7477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:21.177065 7477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.182891 7477 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:36637
I20260924 23:16:21.182953 7605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:36637 every 8 connection(s)
I20260924 23:16:21.183269 7477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:16:21.187077 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 7477
I20260924 23:16:21.187167 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:16:21.187302 1043 external_mini_cluster.cc:1428] Setting key d8e427c3f86263f75360112e292dc1ef
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291781, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM
I20260924 23:16:21.191944 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.184010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:59963 (local address 127.1.4.254:42079)
0924 23:16:21.184063 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.184065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.184250 (+ 185us) server_negotiation.cc:407] Connection header received
0924 23:16:21.184704 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.184705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.184715 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.184780 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.185704 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.186214 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.187193 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.187483 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.187540 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.189691 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.189692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.189692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.189699 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.190772 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.191117 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.191118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.191118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.191163 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.191347 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.191348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.191348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.191486 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.191536 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.191819 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.191879 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 23:16:21.192157 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.183943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.195:59963)
0924 23:16:21.184103 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.184146 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.184621 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.184837 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.184841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.185404 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.185614 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.185619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.186294 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.186294 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.187101 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.187103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.187568 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.187569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.187706 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.188197 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.188205 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.189601 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.190835 (+ 1234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.190841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.190842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.191059 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.191209 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.191209 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.191209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.191306 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.191569 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.191575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.191704 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.191803 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.191941 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260924 23:16:21.192652 7606 heartbeater.cc:347] Connected to a master server at 127.1.4.254:42079
I20260924 23:16:21.192740 7606 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:21.192888 7606 heartbeater.cc:511] Master 127.1.4.254:42079 requested a full tablet report, sending...
I20260924 23:16:21.193223 7146 ts_manager.cc:194] Registered new tserver with Master: 3eec19b8e01544c7a91a6ab99e738b73 (127.1.4.195:36637)
I20260924 23:16:21.193583 7146 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:59963
I20260924 23:16:21.198900 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.4.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.4.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.4.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.253.keytab.
Entry for principal oryx/127.1.4.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.253.keytab.
I20260924 23:16:21.222487 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.253:45945
--webserver_interface=127.1.4.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.253.keytab
--principal=oryx/127.1.4.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:42079,127.1.4.253:45945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:21.301004 7611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:21.301213 7611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:21.301241 7611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:21.302376 7611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:21.302421 7611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:21.302439 7611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:21.302464 7611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:21.302479 7611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:21.303987 7611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:42079,127.1.4.253:45945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.253.keytab
--principal=oryx/127.1.4.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.253:45945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.4.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
W20260924 23:16:21.304095 7611 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.4.254:42079,127.1.4.253:45945'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 23:16:21.304255 7611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:21.304467 7611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:21.309262 7619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:21.309280 7617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:21.309252 7616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:21.309549 7611 server_base.cc:1034] running on GCE node
I20260924 23:16:21.309760 7611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:21.310042 7611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:21.311187 7611 hybrid_clock.cc:648] HybridClock initialized: now 1790291781311176 us; error 32 us; skew 500 ppm
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291781, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:21.313651 7611 init.cc:377] Logged in from keytab as oryx/127.1.4.253@KRBTEST.COM (short username oryx)
I20260924 23:16:21.314103 7611 webserver.cc:460] Webserver started at http://127.1.4.253:33433/ using document root <none> and password file <none>
I20260924 23:16:21.314318 7611 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:21.314366 7611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:21.314502 7611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:21.315603 7611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/data/instance:
uuid: "b69cbd28a4394c14aaff8bb3901f5472"
format_stamp: "Formatted at 2026-09-24 23:16:21 on dist-test-slave-s1hw"
server_key: "91daa15833a0a8b12f5626ee7a8ca57a"
server_key_iv: "5427406c791a6b4f5c9ac8c3eb5fae9a"
server_key_version: "encryptionkey@0"
I20260924 23:16:21.315930 7611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/wal/instance:
uuid: "b69cbd28a4394c14aaff8bb3901f5472"
format_stamp: "Formatted at 2026-09-24 23:16:21 on dist-test-slave-s1hw"
server_key: "91daa15833a0a8b12f5626ee7a8ca57a"
server_key_iv: "5427406c791a6b4f5c9ac8c3eb5fae9a"
server_key_version: "encryptionkey@0"
I20260924 23:16:21.317253 7611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:21.318167 7626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.318367 7611 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:21.318454 7611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/wal
uuid: "b69cbd28a4394c14aaff8bb3901f5472"
format_stamp: "Formatted at 2026-09-24 23:16:21 on dist-test-slave-s1hw"
server_key: "91daa15833a0a8b12f5626ee7a8ca57a"
server_key_iv: "5427406c791a6b4f5c9ac8c3eb5fae9a"
server_key_version: "encryptionkey@0"
I20260924 23:16:21.318513 7611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:21.329368 7611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:21.331156 7611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:21.331291 7611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:21.334972 7611 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.253:45945
I20260924 23:16:21.335044 7678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.253:45945 every 8 connection(s)
I20260924 23:16:21.335345 7611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/data/info.pb
I20260924 23:16:21.336323 7679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:21.336694 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 7611
I20260924 23:16:21.338089 7679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.4.253@KRBTEST.COM
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291781, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.253@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM
I20260924 23:16:21.346783 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.338772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45952 (local address 127.1.4.254:42079)
0924 23:16:21.338821 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.338823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.338963 (+ 140us) server_negotiation.cc:407] Connection header received
0924 23:16:21.339059 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.339060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.339072 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.339118 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.339647 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.340314 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.341321 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.341587 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.341637 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.344160 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.344162 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.344162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.344169 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.345705 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.346019 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.346020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.346021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.346073 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.346244 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.346244 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.346244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.346367 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.346421 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.346616 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.346693 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 23:16:21.346845 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.337311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45458 (local address 127.1.4.253:45945)
0924 23:16:21.337506 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.337517 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.337550 (+ 33us) server_negotiation.cc:407] Connection header received
0924 23:16:21.337982 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.337989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.338036 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.338158 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.338834 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.339763 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.340644 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.341189 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.341293 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.343605 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.343611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.343611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.343617 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.345325 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.345753 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.345756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.345757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.345838 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.346011 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.346012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.346012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.346309 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.346443 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.346530 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.346661 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260924 23:16:21.347007 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.338714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.0.0.1:45952)
0924 23:16:21.338865 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.338908 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.338961 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.339152 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.339155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.339441 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.339559 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.339565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.340406 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.340407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.341254 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.341256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.341659 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.341660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.341732 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.342097 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.342105 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.344082 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.345775 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.345781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.345783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.345956 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.346133 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.346134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.346134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.346187 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.346426 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.346431 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.346508 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.346581 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.346803 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":102,"thread_start_us":71,"threads_started":1}
I20260924 23:16:21.347673 7679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291781, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.253@KRBTEST.COM for oryx/127.1.4.253@KRBTEST.COM
I20260924 23:16:21.353181 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.347897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:45945 (local address 127.0.0.1:45460)
0924 23:16:21.347980 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.347989 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.348019 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.348116 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.348117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.348218 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.348273 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.348274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.348829 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.348829 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.349330 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.349332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.349611 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.349611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.349661 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.350003 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.350008 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.351038 (+ 1030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.352225 (+ 1187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.352226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.352226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.352430 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.352678 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.352679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.352679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.352732 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.352980 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.352981 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.353011 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.353052 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.353121 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 23:16:21.353240 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.347945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45460 (local address 127.1.4.253:45945)
0924 23:16:21.348043 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.348044 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.348050 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:16:21.348064 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.348065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.348072 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.348113 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.348329 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.348774 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.349386 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.349564 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.349608 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.351106 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.351107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.351107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.351112 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.352152 (+ 1040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.352544 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.352545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.352546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.352614 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.352782 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.352783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.352783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.352933 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.352984 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.353074 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.353164 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":82}
I20260924 23:16:21.354521 7679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: Bootstrap starting.
I20260924 23:16:21.355337 7679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:21.355600 7679 log.cc:824] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:21.356334 7679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: No bootstrap required, opened a new log
I20260924 23:16:21.357676 7679 raft_consensus.cc:348] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } }
I20260924 23:16:21.357800 7679 raft_consensus.cc:374] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:21.357820 7679 raft_consensus.cc:729] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69cbd28a4394c14aaff8bb3901f5472, State: Initialized, Role: FOLLOWER
I20260924 23:16:21.357926 7679 consensus_queue.cc:260] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } }
I20260924 23:16:21.358170 7679 sys_catalog.cc:564] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:21.358146 7685 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } } }
I20260924 23:16:21.358225 7685 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 23:16:21.359675 7696 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:16:21.359735 7696 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:16:21.361294 7611 master_runner.cc:428] Detected that this master b69cbd28a4394c14aaff8bb3901f5472 is joining an existing cluster
I20260924 23:16:21.361339 7611 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.4.253:45945
I20260924 23:16:21.368490 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.362331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.1.4.253:45945)
0924 23:16:21.362366 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.362368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.362588 (+ 220us) server_negotiation.cc:407] Connection header received
0924 23:16:21.362646 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.362646 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.362655 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.362695 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.363103 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.363891 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.364953 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.365190 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.365245 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.366134 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.366135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.366135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.366141 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.367517 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.367812 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.367813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.367813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.367864 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.368027 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.368027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.368028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.368131 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.368183 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.368310 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.368393 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:16:21.368572 7703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.362413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:45945 (local address 127.0.0.1:45472)
0924 23:16:21.362538 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.362546 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.362605 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.362697 (+ 92us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.362697 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.362954 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.363035 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.363037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.363970 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.363971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.364878 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.364880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.365279 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.365280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.365319 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.365743 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.365747 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.366085 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.367589 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.367590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.367590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.367752 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.367918 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.367919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.367919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.367979 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.368180 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.368181 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.368227 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.368285 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.368457 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":116,"thread_start_us":81,"threads_started":1}
I20260924 23:16:21.368920 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.362350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.1.4.254:42079)
0924 23:16:21.362393 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.362396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.362516 (+ 120us) server_negotiation.cc:407] Connection header received
0924 23:16:21.362608 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.362609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.362621 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.362682 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.363136 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.363931 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.364700 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.364900 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.364953 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.366102 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.366103 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.366103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.366111 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.367749 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.368132 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.368133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.368133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.368188 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.368369 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.368369 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.368370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.368486 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.368526 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.368762 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.368838 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:16:21.369248 7702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.362300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.0.0.1:45954)
0924 23:16:21.362458 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.362468 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.362541 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.362730 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.362731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.363003 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.363073 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.363074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.364008 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.364008 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.364614 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.364616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.365005 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.365005 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.365066 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.365536 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.365540 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.366012 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.367835 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.367836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.367837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.368048 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.368253 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.368254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.368254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.368306 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.368615 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.368616 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.368657 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.369047 (+ 390us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.369149 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":23,"thread_start_us":96,"threads_started":1}
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291780, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.254@KRBTEST.COM for oryx/127.1.4.253@KRBTEST.COM
I20260924 23:16:21.378412 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.371485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45482 (local address 127.1.4.253:45945)
0924 23:16:21.371520 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.371522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.371684 (+ 162us) server_negotiation.cc:407] Connection header received
0924 23:16:21.371766 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.371767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.371774 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.371816 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.372299 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.372761 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.373843 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.374019 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.374065 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.376136 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.376137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.376138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.376144 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.377401 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.377727 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.377728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.377729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.377780 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.377935 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.377935 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.377936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.378043 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.378095 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.378254 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.378325 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:16:21.378436 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.371440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:45945 (local address 127.0.0.1:45482)
0924 23:16:21.371576 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.371622 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.371695 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.371822 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.371825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.372155 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.372212 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.372214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.372830 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.372831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.373773 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.373776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.374075 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.374076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.374150 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.374528 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.374533 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.376061 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.377476 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.377482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.377484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.377665 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.377829 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.377830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.377830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.377886 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.378094 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.378101 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.378153 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.378223 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.378334 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260924 23:16:21.378937 7146 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.4.253:45945: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472"
member_type: NON_VOTER
last_known_addr {
host: "127.1.4.253"
port: 45945
}
attrs {
promote: true
}
}
dest_uuid: "c3a137b4a02d44709e97e6af2f7dd626"
cas_config_opid_index: -1
I20260924 23:16:21.379226 7146 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: NON_VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: true } }
I20260924 23:16:21.379671 7706 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b69cbd28a4394c14aaff8bb3901f5472 (127.1.4.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: NON_VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: true } } }
I20260924 23:16:21.379931 7653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:21.380681 7706 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.4.253:45945
I20260924 23:16:21.380817 7653 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 1 FOLLOWER]: Refusing update from remote peer c3a137b4a02d44709e97e6af2f7dd626: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 23:16:21.381027 7611 master.cc:537] Master@127.1.4.253:45945 shutting down...
I20260924 23:16:21.381031 7706 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c3a137b4a02d44709e97e6af2f7dd626" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: NON_VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: true } } }
I20260924 23:16:21.381100 7706 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:21.381865 7706 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: NON_VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 23:16:21.382606 7133 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 23:16:21.382992 7133 consensus_peers.cc:487] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 -> Peer b69cbd28a4394c14aaff8bb3901f5472 (127.1.4.253:45945): Couldn't send request to peer b69cbd28a4394c14aaff8bb3901f5472. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 23:16:21.383704 7611 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:21.383792 7611 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:21.383841 7611 tablet_replica.cc:330] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: stopping tablet replica
I20260924 23:16:21.396255 7611 master.cc:552] Master@127.1.4.253:45945 shutdown complete.
I20260924 23:16:21.396309 7611 master_runner.cc:305] Clearing existing system tablet
I20260924 23:16:21.396438 7611 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:21.396481 7611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:21.397146 7611 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:21.397560 7710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.397845 7611 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:21.397951 7611 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:16:21.397979 7611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/wal
uuid: "b69cbd28a4394c14aaff8bb3901f5472"
format_stamp: "Formatted at 2026-09-24 23:16:21 on dist-test-slave-s1hw"
server_key: "91daa15833a0a8b12f5626ee7a8ca57a"
server_key_iv: "5427406c791a6b4f5c9ac8c3eb5fae9a"
server_key_version: "encryptionkey@0"
I20260924 23:16:21.398608 7611 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:21.400003 7611 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 23:16:21.400058 7611 log.cc:1196] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 23:16:21.400218 7611 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: Deleting consensus metadata
I20260924 23:16:21.400309 7611 master_runner.cc:315] Copying system tablet from 127.1.4.254:42079
I20260924 23:16:21.400856 7611 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: tablet copy: Beginning tablet copy session from remote peer at address 127.1.4.254:42079
I20260924 23:16:21.406787 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.401148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.0.0.1:45964)
0924 23:16:21.401253 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.401264 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.401328 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.401453 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.401453 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.401591 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.401645 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.401647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.402588 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.402589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.403169 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.403171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.403634 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.403636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.403693 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.404151 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.404157 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.404443 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.405990 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.405991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.405991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.406143 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.406317 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.406317 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.406317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.406358 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.406563 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.406563 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.406597 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.406650 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.406718 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":66,"threads_started":1}
I20260924 23:16:21.406852 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.401206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45964 (local address 127.1.4.254:42079)
0924 23:16:21.401256 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.401258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.401318 (+ 60us) server_negotiation.cc:407] Connection header received
0924 23:16:21.401373 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.401374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.401385 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.401448 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.401707 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.402504 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.403242 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.403564 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.403631 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.404519 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.404521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.404521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.404527 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.405908 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.406204 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.406205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.406206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.406261 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.406390 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.406391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.406392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.406508 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.406565 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.406689 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.406759 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:16:21.407336 7166 tablet_copy_service.cc:140] P c3a137b4a02d44709e97e6af2f7dd626: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b69cbd28a4394c14aaff8bb3901f5472 ({username='oryx', principal='oryx/127.1.4.253@KRBTEST.COM'} at 127.0.0.1:45964)
I20260924 23:16:21.407449 7166 tablet_copy_service.cc:161] P c3a137b4a02d44709e97e6af2f7dd626: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b69cbd28a4394c14aaff8bb3901f5472 at {username='oryx', principal='oryx/127.1.4.253@KRBTEST.COM'} at 127.0.0.1:45964: session id = b69cbd28a4394c14aaff8bb3901f5472-00000000000000000000000000000000
I20260924 23:16:21.408118 7166 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 23:16:21.408695 7611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 23:16:21.409822 7611 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: tablet copy: Starting download of 0 data blocks...
I20260924 23:16:21.409966 7611 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: tablet copy: Starting download of 1 WAL segments...
I20260924 23:16:21.410741 7611 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 23:16:21.411605 7166 tablet_copy_service.cc:342] P c3a137b4a02d44709e97e6af2f7dd626: Request end of tablet copy session b69cbd28a4394c14aaff8bb3901f5472-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.4.253@KRBTEST.COM'} at 127.0.0.1:45964
I20260924 23:16:21.411661 7166 tablet_copy_service.cc:433] P c3a137b4a02d44709e97e6af2f7dd626: ending tablet copy session b69cbd28a4394c14aaff8bb3901f5472-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b69cbd28a4394c14aaff8bb3901f5472
I20260924 23:16:21.413689 7611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:21.414914 7718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:21.415103 7717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:21.415153 7720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:21.415488 7611 server_base.cc:1034] running on GCE node
I20260924 23:16:21.415580 7611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:21.415709 7611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:21.416836 7611 hybrid_clock.cc:648] HybridClock initialized: now 1790291781416823 us; error 30 us; skew 500 ppm
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291781, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:21.418107 7611 init.cc:377] Logged in from keytab as oryx/127.1.4.253@KRBTEST.COM (short username oryx)
I20260924 23:16:21.418361 7611 webserver.cc:460] Webserver started at http://127.1.4.253:44787/ using document root <none> and password file <none>
I20260924 23:16:21.418493 7611 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:21.418526 7611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:21.419184 7611 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:21.419689 7727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.419800 7611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.419844 7611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/wal
uuid: "b69cbd28a4394c14aaff8bb3901f5472"
format_stamp: "Formatted at 2026-09-24 23:16:21 on dist-test-slave-s1hw"
server_key: "91daa15833a0a8b12f5626ee7a8ca57a"
server_key_iv: "5427406c791a6b4f5c9ac8c3eb5fae9a"
server_key_version: "encryptionkey@0"
I20260924 23:16:21.419900 7611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:21.438236 7611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:21.439706 7611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:21.442243 7611 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.253:45945
I20260924 23:16:21.442281 7779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.253:45945 every 8 connection(s)
I20260924 23:16:21.442538 7611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-1/data/info.pb
I20260924 23:16:21.443181 7780 sys_catalog.cc:263] Verifying existing consensus state
I20260924 23:16:21.443621 7780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: Bootstrap starting.
I20260924 23:16:21.445847 7780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:16:21.446060 7780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: Bootstrap complete.
I20260924 23:16:21.446259 7780 raft_consensus.cc:348] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: NON_VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: true } }
I20260924 23:16:21.446329 7780 raft_consensus.cc:729] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b69cbd28a4394c14aaff8bb3901f5472, State: Initialized, Role: LEARNER
I20260924 23:16:21.446377 7780 consensus_queue.cc:260] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: NON_VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: true } }
I20260924 23:16:21.446527 7780 sys_catalog.cc:564] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:21.446537 7783 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: NON_VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: true } } }
I20260924 23:16:21.446633 7783 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: This master's current role is: LEARNER
I20260924 23:16:21.447822 7794 catalog_manager.cc:1259] Loaded cluster ID: 79e249b6f5f4479fa2bb7f5060558ed9
I20260924 23:16:21.447860 7794 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: loading cluster ID for follower catalog manager: success
I20260924 23:16:21.448849 7794 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: acquiring CA information for follower catalog manager: success
I20260924 23:16:21.449120 7794 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 23:16:21.455524 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.448447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.1.4.253:45945)
0924 23:16:21.448556 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.448558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.448566 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:21.448590 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.448591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.448601 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.449325 (+ 724us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.449357 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.450361 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.451095 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.451441 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.451513 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.452660 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.452661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.452661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.452668 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.454336 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.454664 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.454665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.454665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.454729 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.454882 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.454883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.454884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.455023 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.455119 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.455301 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.455418 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":46,"threads_started":1}
WARNING: no policy specified for oryx/127.1.4.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.4.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.4.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.252.keytab.
Entry for principal oryx/127.1.4.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.252.keytab.
I20260924 23:16:21.490942 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.252:45637
--webserver_interface=127.1.4.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.252.keytab
--principal=oryx/127.1.4.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:42079,127.1.4.253:45945,127.1.4.252:45637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:21.572863 7798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:21.573092 7798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:21.573123 7798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:21.574343 7798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:21.574391 7798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:21.574409 7798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:21.574425 7798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:21.574440 7798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:21.576146 7798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:42079,127.1.4.253:45945,127.1.4.252:45637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.252.keytab
--principal=oryx/127.1.4.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.252:45637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.4.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:21.576421 7798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:21.576651 7798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:21.581529 7803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:21.581612 7798 server_base.cc:1034] running on GCE node
W20260924 23:16:21.581568 7806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:21.581548 7804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:21.582046 7798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:21.582365 7798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:21.583510 7798 hybrid_clock.cc:648] HybridClock initialized: now 1790291781583491 us; error 36 us; skew 500 ppm
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291781, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:21.586081 7798 init.cc:377] Logged in from keytab as oryx/127.1.4.252@KRBTEST.COM (short username oryx)
I20260924 23:16:21.586535 7798 webserver.cc:460] Webserver started at http://127.1.4.252:35303/ using document root <none> and password file <none>
I20260924 23:16:21.586760 7798 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:21.586807 7798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:21.586933 7798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:21.588021 7798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/data/instance:
uuid: "5c7fbf9789ba4b5fb7c52c61869193ea"
format_stamp: "Formatted at 2026-09-24 23:16:21 on dist-test-slave-s1hw"
server_key: "0bfa9a66d0c78add2bbcaeee03295269"
server_key_iv: "54cc6b32bf3cd7919179eb598f96eba3"
server_key_version: "encryptionkey@0"
I20260924 23:16:21.588354 7798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/wal/instance:
uuid: "5c7fbf9789ba4b5fb7c52c61869193ea"
format_stamp: "Formatted at 2026-09-24 23:16:21 on dist-test-slave-s1hw"
server_key: "0bfa9a66d0c78add2bbcaeee03295269"
server_key_iv: "54cc6b32bf3cd7919179eb598f96eba3"
server_key_version: "encryptionkey@0"
I20260924 23:16:21.589555 7798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:16:21.590257 7813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.590411 7798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:21.590490 7798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/wal
uuid: "5c7fbf9789ba4b5fb7c52c61869193ea"
format_stamp: "Formatted at 2026-09-24 23:16:21 on dist-test-slave-s1hw"
server_key: "0bfa9a66d0c78add2bbcaeee03295269"
server_key_iv: "54cc6b32bf3cd7919179eb598f96eba3"
server_key_version: "encryptionkey@0"
I20260924 23:16:21.590551 7798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:21.612730 7798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:21.614982 7798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:21.615125 7798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:21.618783 7798 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.252:45637
I20260924 23:16:21.618827 7865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.252:45637 every 8 connection(s)
I20260924 23:16:21.619179 7798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/data/info.pb
I20260924 23:16:21.620177 7866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:21.621690 7866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:16:21.625896 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 7798
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291781, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.252@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM
I20260924 23:16:21.630355 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.622418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.1.4.254:42079)
0924 23:16:21.622473 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.622475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.622569 (+ 94us) server_negotiation.cc:407] Connection header received
0924 23:16:21.623025 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.623026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.623036 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.623094 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.623722 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.624237 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.625238 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.625434 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.625492 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.627659 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.627661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.627662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.627670 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.629026 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.629431 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.629432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.629432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.629522 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.629773 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.629774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.629774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.629872 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.629920 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.630231 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.630290 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 23:16:21.630611 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.622303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.0.0.1:45976)
0924 23:16:21.622455 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.622496 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.622921 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.623150 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.623153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.623417 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.623620 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.623624 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.624329 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.624329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.625152 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.625154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.625497 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.625498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.625592 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.626107 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.626118 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.627585 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.629101 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.629106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.629108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.629358 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.629582 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.629583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.629583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.629711 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.629928 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.629933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.630100 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.630220 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.630374 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260924 23:16:21.631136 7866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.4.252@KRBTEST.COM
I20260924 23:16:21.635284 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.626355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.1.4.252:45637)
0924 23:16:21.626530 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.626533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.626547 (+ 14us) server_negotiation.cc:407] Connection header received
0924 23:16:21.626564 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.626565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.626580 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.626640 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.627172 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.628049 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.628708 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.629172 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.629217 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.631814 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.631821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.631822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.631829 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.633560 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.633997 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.634000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.634001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.634058 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.634311 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.634312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.634312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.634497 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.635049 (+ 552us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.635096 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.635171 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":87,"threads_started":1}
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291781, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.252@KRBTEST.COM for oryx/127.1.4.253@KRBTEST.COM
I20260924 23:16:21.638832 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.631390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:45945 (local address 127.0.0.1:45492)
0924 23:16:21.631440 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.631451 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.631489 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.631732 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.631733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.631877 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.631927 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.631928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.632847 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.632849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.633628 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.633630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.634207 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.634208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.634277 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.634646 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.634651 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.635998 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.637857 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.637858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.637859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.638023 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.638200 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.638200 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.638201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.638251 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.638518 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.638519 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.638561 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.638648 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.638736 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 23:16:21.638885 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.631524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.1.4.253:45945)
0924 23:16:21.631608 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.631610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.631616 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:16:21.631641 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.631642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.631652 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.631693 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.632006 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.632721 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.633758 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.634068 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.634141 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.636117 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.636119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.636119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.636126 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.637786 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.638081 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.638082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.638083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.638134 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.638301 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.638302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.638302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.638423 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.638493 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.638706 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.638798 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 23:16:21.639179 7866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291781, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.252@KRBTEST.COM for oryx/127.1.4.252@KRBTEST.COM
I20260924 23:16:21.644829 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.639439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:45637 (local address 127.0.0.1:34864)
0924 23:16:21.639467 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.639476 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.639528 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.639623 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.639624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.639745 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.639776 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.639778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.640667 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.640668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.641205 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.641207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.641522 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.641523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.641565 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.641842 (+ 277us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.641845 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.642842 (+ 997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.643985 (+ 1143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.643986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.643986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.644140 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.644286 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.644287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.644287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.644335 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.644529 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.644530 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.644565 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.644624 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.644710 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260924 23:16:21.644871 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.639450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34864 (local address 127.1.4.252:45637)
0924 23:16:21.639481 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.639482 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.639514 (+ 32us) server_negotiation.cc:407] Connection header received
0924 23:16:21.639566 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.639566 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.639575 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.639613 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.639876 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.640595 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.641270 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.641470 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.641522 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.642907 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.642908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.642909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.642915 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.643922 (+ 1007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.644194 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.644195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.644196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.644239 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.644383 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.644383 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.644384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.644474 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.644529 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.644649 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.644755 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":19}
I20260924 23:16:21.646100 7866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: Bootstrap starting.
I20260924 23:16:21.646941 7866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:21.647156 7866 log.cc:824] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:21.647825 7866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: No bootstrap required, opened a new log
I20260924 23:16:21.649552 7866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } }
I20260924 23:16:21.649681 7866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:21.649703 7866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7fbf9789ba4b5fb7c52c61869193ea, State: Initialized, Role: FOLLOWER
I20260924 23:16:21.649801 7866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } }
I20260924 23:16:21.650070 7866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:21.650032 7871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } } }
I20260924 23:16:21.650125 7871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [sys.catalog]: This master's current role is: FOLLOWER
W20260924 23:16:21.651575 7882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:16:21.651634 7882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:16:21.653661 7798 master_runner.cc:428] Detected that this master 5c7fbf9789ba4b5fb7c52c61869193ea is joining an existing cluster
I20260924 23:16:21.653699 7798 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.4.252:45637
I20260924 23:16:21.660836 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.654551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.1.4.254:42079)
0924 23:16:21.654592 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.654594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.654747 (+ 153us) server_negotiation.cc:407] Connection header received
0924 23:16:21.654841 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.654842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.654854 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.654884 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.655452 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.656255 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.657312 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.657594 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.657626 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.658569 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.658571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.658571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.658578 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.659883 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.660171 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.660172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.660172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.660224 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.660380 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.660381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.660381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.660488 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.660542 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.660681 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.660746 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:16:21.660851 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.654522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.0.0.1:45978)
0924 23:16:21.654693 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.654703 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.654771 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.654954 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.654955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.655283 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.655350 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.655352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.656354 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.656355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.657232 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.657233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.657702 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.657702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.657756 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.658159 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.658164 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.658491 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.659951 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.659952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.659953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.660112 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.660273 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.660273 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.660274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.660333 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.660538 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.660539 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.660594 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.660651 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.660746 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":24,"thread_start_us":114,"threads_started":1}
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291780, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.254@KRBTEST.COM for oryx/127.1.4.252@KRBTEST.COM
I20260924 23:16:21.679322 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.662692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:45637 (local address 127.0.0.1:34890)
0924 23:16:21.662741 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.662750 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.662782 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.663006 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.663006 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.663161 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.663203 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.663205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.663894 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.663895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.664710 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.664712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.665064 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.665065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.665129 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.665503 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.665507 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.666752 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.677370 (+ 10618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.677372 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.677373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.677598 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.678340 (+ 742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.678341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.678342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.678411 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.679012 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.679013 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.679060 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.679107 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.679216 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 23:16:21.679497 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.654834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:45637 (local address 127.0.0.1:34876)
0924 23:16:21.654979 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.654987 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.655042 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.655166 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.655167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.655383 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.655437 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.655439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.656354 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.656355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.657228 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.657230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.657700 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.657701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.657757 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.675930 (+ 18173us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.675936 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.676826 (+ 890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.678478 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.678479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.678480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.678677 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.678884 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.678884 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.678885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.678933 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.679173 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.679173 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.679206 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.679272 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.679385 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":24,"thread_start_us":88,"threads_started":1}
I20260924 23:16:21.679735 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.654776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34876 (local address 127.1.4.252:45637)
0924 23:16:21.654805 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.654807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.655025 (+ 218us) server_negotiation.cc:407] Connection header received
0924 23:16:21.655095 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.655095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.655102 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.655124 (+ 22us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.655511 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.656253 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.657310 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.657590 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.657625 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.676945 (+ 19320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.676946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.676947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.676953 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.678407 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.678749 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.678750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.678751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.678815 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.678994 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.678995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.678995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.679104 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.679154 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.679606 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.679669 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 23:16:21.680150 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.654547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.1.4.253:45945)
0924 23:16:21.654587 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.654589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.654810 (+ 221us) server_negotiation.cc:407] Connection header received
0924 23:16:21.654909 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.654910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.654920 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.654948 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.655622 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.656364 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.657417 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.657708 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.657747 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.677056 (+ 19309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.677058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.677058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.677066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.678985 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.679306 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.679307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.679308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.679357 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.679636 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.679637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.679638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.679759 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.679799 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.679992 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.680068 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:16:21.680441 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.662743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.1.4.252:45637)
0924 23:16:21.662897 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.662899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.662905 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:16:21.662931 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.662931 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.662939 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.662993 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.663261 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.663824 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.664787 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.664995 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.665061 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.666824 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.666826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.666826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.666833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.677293 (+ 10460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.677681 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.677682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.677683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.677741 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.678759 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.678760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.678760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.678933 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.680197 (+ 1264us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.680236 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.680314 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260924 23:16:21.680522 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.654623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:45945 (local address 127.0.0.1:45496)
0924 23:16:21.654748 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.654756 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.654833 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.655010 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.655011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.655330 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.655522 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.655524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.656455 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.656456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.657314 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.657317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.657819 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.657819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.657896 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.676151 (+ 18255us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.676157 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.676934 (+ 777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.679059 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.679060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.679061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.679233 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.679444 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.679445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.679445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.679505 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.679848 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.679849 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.679886 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.680262 (+ 376us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.680366 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":2,"thread_start_us":68,"threads_started":1}
I20260924 23:16:21.680861 7146 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.4.252:45637: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea"
member_type: NON_VOTER
last_known_addr {
host: "127.1.4.252"
port: 45637
}
attrs {
promote: true
}
}
dest_uuid: "c3a137b4a02d44709e97e6af2f7dd626"
cas_config_opid_index: 5
I20260924 23:16:21.681100 7146 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: NON_VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: true } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } }
I20260924 23:16:21.681401 7705 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5c7fbf9789ba4b5fb7c52c61869193ea (127.1.4.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: NON_VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: true } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } } }
I20260924 23:16:21.682356 7705 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.4.252:45637
I20260924 23:16:21.682436 7705 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c3a137b4a02d44709e97e6af2f7dd626" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: NON_VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: true } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } } }
I20260924 23:16:21.682504 7705 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:21.682690 7798 master.cc:537] Master@127.1.4.252:45637 shutting down...
W20260924 23:16:21.683197 7131 consensus_peers.cc:487] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 -> Peer 5c7fbf9789ba4b5fb7c52c61869193ea (127.1.4.252:45637): Couldn't send request to peer 5c7fbf9789ba4b5fb7c52c61869193ea. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 23:16:21.685330 7798 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:21.685422 7798 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:21.685472 7798 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: stopping tablet replica
I20260924 23:16:21.686529 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.682731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:45945 (local address 127.0.0.1:45510)
0924 23:16:21.682974 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.682984 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.683022 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.683218 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.683219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.683220 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:21.683261 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.683263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.684238 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.684239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.685548 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.685551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.686298 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.686299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.686364 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.686387 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.686431 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.686468 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260924 23:16:21.686756 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.682778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45510 (local address 127.1.4.253:45945)
0924 23:16:21.682843 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.682845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.683069 (+ 224us) server_negotiation.cc:407] Connection header received
0924 23:16:21.683089 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.683090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.683110 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.683168 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:21.683322 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.684152 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.685643 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.686215 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.686277 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.686345 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.686658 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.686698 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 23:16:21.686950 7754 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 1 LEARNER]: Refusing update from remote peer c3a137b4a02d44709e97e6af2f7dd626: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 23:16:21.687147 7706 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: NON_VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260924 23:16:21.687309 7099 connection.cc:535] client connection to 127.1.4.252:45637 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 23:16:21.687742 7754 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5c7fbf9789ba4b5fb7c52c61869193ea (127.1.4.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: NON_VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: true } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } } }
I20260924 23:16:21.688623 7893 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3a137b4a02d44709e97e6af2f7dd626" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: NON_VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: true } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } } }
I20260924 23:16:21.688700 7893 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: This master's current role is: LEARNER
I20260924 23:16:21.688812 7706 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626: attempting to promote NON_VOTER b69cbd28a4394c14aaff8bb3901f5472 to VOTER
I20260924 23:16:21.688839 7783 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3a137b4a02d44709e97e6af2f7dd626. Latest consensus state: current_term: 1 leader_uuid: "c3a137b4a02d44709e97e6af2f7dd626" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: NON_VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: true } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } } }
I20260924 23:16:21.688881 7783 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: This master's current role is: LEARNER
I20260924 23:16:21.689004 7706 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } }
I20260924 23:16:21.689510 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.689425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:45637 (local address 127.0.0.1:34900)
0924 23:16:21.689455 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.689490 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:45637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 23:16:21.689570 7754 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 1 LEARNER]: Refusing update from remote peer c3a137b4a02d44709e97e6af2f7dd626: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 23:16:21.689764 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.689700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:45637 (local address 127.0.0.1:34904)
0924 23:16:21.689724 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.689745 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:45637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
I20260924 23:16:21.689762 7705 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 23:16:21.689877 7131 consensus_peers.cc:487] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 -> Peer 5c7fbf9789ba4b5fb7c52c61869193ea (127.1.4.252:45637): Couldn't send request to peer 5c7fbf9789ba4b5fb7c52c61869193ea. Status: Network error: Client connection negotiation failed: client connection to 127.1.4.252:45637: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 23:16:21.690371 7894 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b69cbd28a4394c14aaff8bb3901f5472 (127.1.4.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } } }
I20260924 23:16:21.690595 7754 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b69cbd28a4394c14aaff8bb3901f5472 (127.1.4.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } } }
I20260924 23:16:21.691120 7705 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c3a137b4a02d44709e97e6af2f7dd626" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } } }
I20260924 23:16:21.691185 7705 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:21.691255 7894 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3a137b4a02d44709e97e6af2f7dd626" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } } }
I20260924 23:16:21.691298 7894 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:21.691424 7783 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3a137b4a02d44709e97e6af2f7dd626" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } } }
I20260924 23:16:21.691497 7783 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:16:21.698767 7798 master.cc:552] Master@127.1.4.252:45637 shutdown complete.
I20260924 23:16:21.698814 7798 master_runner.cc:305] Clearing existing system tablet
I20260924 23:16:21.698925 7798 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:21.698959 7798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:21.699548 7798 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:16:21.700038 7896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.700176 7798 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:21.700242 7798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.700268 7798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/wal
uuid: "5c7fbf9789ba4b5fb7c52c61869193ea"
format_stamp: "Formatted at 2026-09-24 23:16:21 on dist-test-slave-s1hw"
server_key: "0bfa9a66d0c78add2bbcaeee03295269"
server_key_iv: "54cc6b32bf3cd7919179eb598f96eba3"
server_key_version: "encryptionkey@0"
I20260924 23:16:21.701093 7798 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:21.702620 7798 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 23:16:21.702690 7798 log.cc:1196] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 23:16:21.702873 7798 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: Deleting consensus metadata
I20260924 23:16:21.702982 7798 master_runner.cc:315] Copying system tablet from 127.1.4.254:42079
I20260924 23:16:21.703629 7798 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: tablet copy: Beginning tablet copy session from remote peer at address 127.1.4.254:42079
I20260924 23:16:21.709582 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.703911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.0.0.1:45986)
0924 23:16:21.704048 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.704058 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.704110 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.704246 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.704246 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.704391 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.704450 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.704451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.705325 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.705326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.705952 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.705955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.706406 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.706407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.706449 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.706884 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.706887 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.707207 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.708763 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.708764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.708764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.708919 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.709096 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.709097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.709097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.709142 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.709367 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.709368 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.709402 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.709443 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.709516 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260924 23:16:21.709632 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.703972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45986 (local address 127.1.4.254:42079)
0924 23:16:21.704023 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.704025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.704102 (+ 77us) server_negotiation.cc:407] Connection header received
0924 23:16:21.704153 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.704154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.704164 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.704243 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.704504 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.705241 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.706032 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.706335 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.706412 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.707289 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.707291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.707291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.707298 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.708689 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.708979 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.708980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.708980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.709038 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.709186 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.709187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.709187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.709313 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.709386 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.709467 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.709542 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 23:16:21.709991 7166 tablet_copy_service.cc:140] P c3a137b4a02d44709e97e6af2f7dd626: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5c7fbf9789ba4b5fb7c52c61869193ea ({username='oryx', principal='oryx/127.1.4.252@KRBTEST.COM'} at 127.0.0.1:45986)
I20260924 23:16:21.710052 7166 tablet_copy_service.cc:161] P c3a137b4a02d44709e97e6af2f7dd626: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5c7fbf9789ba4b5fb7c52c61869193ea at {username='oryx', principal='oryx/127.1.4.252@KRBTEST.COM'} at 127.0.0.1:45986: session id = 5c7fbf9789ba4b5fb7c52c61869193ea-00000000000000000000000000000000
I20260924 23:16:21.710472 7166 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 23:16:21.711179 7798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 23:16:21.712477 7798 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: tablet copy: Starting download of 0 data blocks...
I20260924 23:16:21.712625 7798 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: tablet copy: Starting download of 1 WAL segments...
I20260924 23:16:21.713375 7798 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 23:16:21.714239 7166 tablet_copy_service.cc:342] P c3a137b4a02d44709e97e6af2f7dd626: Request end of tablet copy session 5c7fbf9789ba4b5fb7c52c61869193ea-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.4.252@KRBTEST.COM'} at 127.0.0.1:45986
I20260924 23:16:21.714298 7166 tablet_copy_service.cc:433] P c3a137b4a02d44709e97e6af2f7dd626: ending tablet copy session 5c7fbf9789ba4b5fb7c52c61869193ea-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5c7fbf9789ba4b5fb7c52c61869193ea
I20260924 23:16:21.716943 7798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:21.718145 7903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:21.718225 7906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:21.718346 7798 server_base.cc:1034] running on GCE node
W20260924 23:16:21.719159 7904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:21.719349 7798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:21.719547 7798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:21.720664 7798 hybrid_clock.cc:648] HybridClock initialized: now 1790291781720653 us; error 30 us; skew 500 ppm
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291781, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:21.721988 7798 init.cc:377] Logged in from keytab as oryx/127.1.4.252@KRBTEST.COM (short username oryx)
I20260924 23:16:21.722244 7798 webserver.cc:460] Webserver started at http://127.1.4.252:46469/ using document root <none> and password file <none>
I20260924 23:16:21.722349 7798 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:21.722389 7798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:21.723032 7798 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:16:21.723552 7913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.723667 7798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.723717 7798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/wal
uuid: "5c7fbf9789ba4b5fb7c52c61869193ea"
format_stamp: "Formatted at 2026-09-24 23:16:21 on dist-test-slave-s1hw"
server_key: "0bfa9a66d0c78add2bbcaeee03295269"
server_key_iv: "54cc6b32bf3cd7919179eb598f96eba3"
server_key_version: "encryptionkey@0"
I20260924 23:16:21.723773 7798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:21.739564 7798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:21.741168 7798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:21.743553 7798 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.252:45637
I20260924 23:16:21.743777 7965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.252:45637 every 8 connection(s)
I20260924 23:16:21.743907 7798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/master-2/data/info.pb
I20260924 23:16:21.744771 7966 sys_catalog.cc:263] Verifying existing consensus state
I20260924 23:16:21.745311 7966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: Bootstrap starting.
I20260924 23:16:21.748631 7966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:16:21.748826 7966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: Bootstrap complete.
I20260924 23:16:21.749010 7966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } }
I20260924 23:16:21.749080 7966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5c7fbf9789ba4b5fb7c52c61869193ea, State: Initialized, Role: LEARNER
I20260924 23:16:21.749116 7966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } }
I20260924 23:16:21.749287 7966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:21.749287 7969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } } }
I20260924 23:16:21.749387 7969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [sys.catalog]: This master's current role is: LEARNER
I20260924 23:16:21.750536 7980 catalog_manager.cc:1259] Loaded cluster ID: 79e249b6f5f4479fa2bb7f5060558ed9
I20260924 23:16:21.750583 7980 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: loading cluster ID for follower catalog manager: success
I20260924 23:16:21.751995 7980 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: acquiring CA information for follower catalog manager: success
I20260924 23:16:21.752305 7980 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 23:16:21.794972 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.788324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34910 (local address 127.1.4.252:45637)
0924 23:16:21.788460 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.788462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.788472 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:16:21.788500 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.788501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.788513 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.788590 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.789142 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.790143 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.790862 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.791195 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.791297 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.792489 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.792490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.792490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.792498 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.794050 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.794345 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.794346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.794347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.794406 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.794540 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.794540 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.794541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.794656 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.794757 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.794795 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.794869 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260924 23:16:21.795526 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 7209
I20260924 23:16:21.800069 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:34339
--local_ip_for_outbound_sockets=127.1.4.193
--tserver_master_addrs=127.1.4.254:42079
--webserver_port=34479
--webserver_interface=127.1.4.193
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.193.keytab
--principal=oryx/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.4.254:42079,127.1.4.253:45945
--tserver_master_addrs=127.1.4.254:42079,127.1.4.253:45945,127.1.4.252:45637 with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:21.880566 7982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:21.880767 7982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:21.880789 7982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:21.881865 7982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:21.881898 7982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:21.881923 7982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:16:21.883302 7982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.193.keytab
--principal=oryx/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:34339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=34479
--tserver_master_addrs=127.1.4.254:42079,127.1.4.253:45945,127.1.4.252:45637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:21.883530 7982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:21.883713 7982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:16:21.888515 7982 server_base.cc:1034] running on GCE node
W20260924 23:16:21.888517 7988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:21.888450 7990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:21.888473 7987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:21.888825 7982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:21.889056 7982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:21.890195 7982 hybrid_clock.cc:648] HybridClock initialized: now 1790291781890182 us; error 28 us; skew 500 ppm
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291781, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:21.892508 7982 init.cc:377] Logged in from keytab as oryx/127.1.4.193@KRBTEST.COM (short username oryx)
I20260924 23:16:21.892885 7982 webserver.cc:460] Webserver started at http://127.1.4.193:34479/ using document root <none> and password file <none>
I20260924 23:16:21.893083 7982 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:21.893142 7982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:21.894429 7982 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:21.894925 7997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.895084 7982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:21.895164 7982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "08ab7c555b634abda9f23607f5ee397c"
format_stamp: "Formatted at 2026-09-24 23:16:20 on dist-test-slave-s1hw"
server_key: "e668dd492baeddca61abe92ddc836890"
server_key_iv: "4ea44b4df3a313bfdbf258b892216129"
server_key_version: "encryptionkey@0"
I20260924 23:16:21.895509 7982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:21.912760 7982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:21.914651 7982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:21.914791 7982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:21.915015 7982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:21.915309 7982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:21.915347 7982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.915395 7982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:21.915413 7982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:21.920893 7982 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:34339
I20260924 23:16:21.920951 8110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:34339 every 8 connection(s)
I20260924 23:16:21.921314 7982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:16:21.924763 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 7982
I20260924 23:16:21.924925 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 7343
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291781, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.253@KRBTEST.COM
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291781, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.252@KRBTEST.COM
Sep 24 23:16:21 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291781, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM
I20260924 23:16:21.934391 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:37679
--local_ip_for_outbound_sockets=127.1.4.194
--tserver_master_addrs=127.1.4.254:42079
--webserver_port=34177
--webserver_interface=127.1.4.194
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.194.keytab
--principal=oryx/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.4.254:42079,127.1.4.253:45945
--tserver_master_addrs=127.1.4.254:42079,127.1.4.253:45945,127.1.4.252:45637 with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:16:21.934511 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.922012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:57381 (local address 127.1.4.253:45945)
0924 23:16:21.922061 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.922063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.922296 (+ 233us) server_negotiation.cc:407] Connection header received
0924 23:16:21.922926 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.922927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.922938 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.922967 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.924748 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.925367 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.926272 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.926534 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.926571 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.930583 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.930584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.930585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.930605 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.932207 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.932725 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.932726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.932727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.932787 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.933132 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.933134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.933134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.933290 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.933331 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.934316 (+ 985us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.934411 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:16:21.934693 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.922017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:45945 (local address 127.1.4.193:57381)
0924 23:16:21.922169 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.922209 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.922813 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.923052 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.923058 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.923554 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.923792 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.923797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.925431 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.925432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.926178 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.926180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.926645 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.926646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.926766 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.928045 (+ 1279us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.928055 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.930511 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.932297 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.932306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.932309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.932627 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.932859 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.932860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.932860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.933020 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.933963 (+ 943us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.933971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.934158 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.934285 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.934460 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":13,"thread_start_us":91,"threads_started":1}
I20260924 23:16:21.935164 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.922061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:54679 (local address 127.1.4.252:45637)
0924 23:16:21.922104 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.922107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.922287 (+ 180us) server_negotiation.cc:407] Connection header received
0924 23:16:21.922906 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.922907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.922922 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.922966 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.923873 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.924668 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.927084 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.927306 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.927353 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.931407 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.931409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.931409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.931417 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.933250 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.933697 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.933698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.933699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.933755 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.933935 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.933936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.933936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.934081 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.934137 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.934974 (+ 837us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.935071 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:16:21.935482 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.922017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:45637 (local address 127.1.4.193:54679)
0924 23:16:21.922168 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.922209 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.922814 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.923340 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.923341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.923559 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.923792 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.923796 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.924773 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.924774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.927010 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.927012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.927422 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.927423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.927482 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.928158 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.928162 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.930314 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.933402 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.933403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.933404 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.933604 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.933823 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.933824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.933824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.933879 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.934798 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.934799 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.934844 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.935241 (+ 397us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.935359 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":1241,"thread_start_us":83,"threads_started":1}
I20260924 23:16:21.935951 8113 heartbeater.cc:347] Connected to a master server at 127.1.4.253:45945
I20260924 23:16:21.935959 8111 heartbeater.cc:347] Connected to a master server at 127.1.4.252:45637
I20260924 23:16:21.936050 8113 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:21.936064 8111 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:21.936249 8113 heartbeater.cc:511] Master 127.1.4.253:45945 requested a full tablet report, sending...
I20260924 23:16:21.936249 8111 heartbeater.cc:511] Master 127.1.4.252:45637 requested a full tablet report, sending...
I20260924 23:16:21.936751 7744 ts_manager.cc:194] Registered new tserver with Master: 08ab7c555b634abda9f23607f5ee397c (127.1.4.193:34339)
I20260924 23:16:21.936751 7930 ts_manager.cc:194] Registered new tserver with Master: 08ab7c555b634abda9f23607f5ee397c (127.1.4.193:34339)
I20260924 23:16:21.939599 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.922397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:59095 (local address 127.1.4.254:42079)
0924 23:16:21.922465 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:21.922467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:21.922625 (+ 158us) server_negotiation.cc:407] Connection header received
0924 23:16:21.922926 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:21.922927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:21.922938 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:21.922967 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:21.923902 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.924648 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.927113 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.927436 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.927473 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.930914 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:21.930916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:21.930916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:21.930998 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:21.938467 (+ 7469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.938849 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.938850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.938850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.938911 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:21.939103 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:21.939104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:21.939104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:21.939219 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:21.939274 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:21.939426 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 23:16:21.939508 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 23:16:21.939625 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:21.922295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.193:59095)
0924 23:16:21.922560 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:21.922570 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:21.922824 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:21.923062 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:21.923062 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:21.923556 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:21.923792 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.923797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.924755 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.924756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.926178 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:21.926180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:21.927551 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:21.927552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:21.927612 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:21.928728 (+ 1116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:21.928733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:21.930791 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:21.938564 (+ 7773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.938565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.938566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.938772 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.938991 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:21.938992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:21.938992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:21.939051 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:21.939272 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:21.939272 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:21.939316 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 23:16:21.939399 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 23:16:21.939509 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":22,"thread_start_us":55,"threads_started":1}
I20260924 23:16:21.939994 8112 heartbeater.cc:347] Connected to a master server at 127.1.4.254:42079
I20260924 23:16:21.940045 8112 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:21.940116 8112 heartbeater.cc:511] Master 127.1.4.254:42079 requested a full tablet report, sending...
I20260924 23:16:21.940372 7146 ts_manager.cc:194] Re-registered known tserver with Master: 08ab7c555b634abda9f23607f5ee397c (127.1.4.193:34339)
I20260924 23:16:21.940743 7146 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:59095
W20260924 23:16:22.018291 8118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:22.018517 8118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:22.018539 8118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:22.019783 8118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:22.019826 8118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:22.019852 8118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:16:22.021327 8118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.194.keytab
--principal=oryx/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:37679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=34177
--tserver_master_addrs=127.1.4.254:42079,127.1.4.253:45945,127.1.4.252:45637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:22.021522 8118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:22.021693 8118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:22.026548 8124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:22.026612 8123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:22.026692 8118 server_base.cc:1034] running on GCE node
W20260924 23:16:22.026554 8126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:22.026962 8118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:22.027202 8118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:22.028354 8118 hybrid_clock.cc:648] HybridClock initialized: now 1790291782028320 us; error 44 us; skew 500 ppm
Sep 24 23:16:22 dist-test-slave-s1hw krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291782, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:22.030655 8118 init.cc:377] Logged in from keytab as oryx/127.1.4.194@KRBTEST.COM (short username oryx)
I20260924 23:16:22.031096 8118 webserver.cc:460] Webserver started at http://127.1.4.194:34177/ using document root <none> and password file <none>
I20260924 23:16:22.031288 8118 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:22.031347 8118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:22.032665 8118 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:22.033176 8133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:22.033329 8118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:22.033416 8118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "e85bb432f60642fb92fa4c620b6ac7a0"
format_stamp: "Formatted at 2026-09-24 23:16:20 on dist-test-slave-s1hw"
server_key: "a11046435b1c8ff1095b3ddb13148bd0"
server_key_iv: "75e4ca3b23e2642c567baaf4fe69d1c1"
server_key_version: "encryptionkey@0"
I20260924 23:16:22.033777 8118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:22.051313 8118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:22.053188 8118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:22.053320 8118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:22.053537 8118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:22.053820 8118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:22.053853 8118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:22.053884 8118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:22.053903 8118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:22.059491 8118 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:37679
I20260924 23:16:22.059545 8246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:37679 every 8 connection(s)
I20260924 23:16:22.059872 8118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:16:22.061326 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 8118
I20260924 23:16:22.061457 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 7477
Sep 24 23:16:22 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291782, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.252@KRBTEST.COM
I20260924 23:16:22.068324 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:36637
--local_ip_for_outbound_sockets=127.1.4.195
--tserver_master_addrs=127.1.4.254:42079
--webserver_port=43247
--webserver_interface=127.1.4.195
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.195.keytab
--principal=oryx/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.4.254:42079,127.1.4.253:45945
--tserver_master_addrs=127.1.4.254:42079,127.1.4.253:45945,127.1.4.252:45637 with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 23:16:22 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291782, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.253@KRBTEST.COM
Sep 24 23:16:22 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291782, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM
I20260924 23:16:22.073025 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.060987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:45265 (local address 127.1.4.252:45637)
0924 23:16:22.061030 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.061031 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.061061 (+ 30us) server_negotiation.cc:407] Connection header received
0924 23:16:22.061937 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.061938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.061951 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.061979 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:22.063003 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.063796 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.065444 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.065744 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.065783 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.069599 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:22.069601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:22.069602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:22.069610 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:22.071438 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.072002 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.072003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.072003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.072066 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.072367 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.072368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.072369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.072507 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:22.072549 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.072835 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.072925 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:16:22.073025 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.060591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:43725 (local address 127.1.4.253:45945)
0924 23:16:22.060644 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.060646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.060909 (+ 263us) server_negotiation.cc:407] Connection header received
0924 23:16:22.061512 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.061513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.061525 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.061556 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:22.062989 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.063772 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.065439 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.065735 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.065775 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.069553 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:22.069554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:22.069555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:22.069563 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:22.071421 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.072047 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.072048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.072048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.072101 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.072295 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.072296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.072296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.072431 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:22.072486 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.072832 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.072918 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 23:16:22.073081 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.061032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:37735 (local address 127.1.4.254:42079)
0924 23:16:22.061074 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.061076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.061085 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:22.062195 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.062196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.062209 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.062237 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:22.063018 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.063796 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.065455 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.065749 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.065783 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.069582 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:22.069584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:22.069584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:22.069591 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:22.071624 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.072344 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.072345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.072345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.072396 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.072593 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.072594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.072594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.072711 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:22.072752 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.072932 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.073005 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:16:22.073643 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.060902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:45637 (local address 127.1.4.194:45265)
0924 23:16:22.061015 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:22.061024 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:22.061470 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:22.062056 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:22.062060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:22.062569 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:22.062879 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.062886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.063930 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.063931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.065346 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.065349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.065860 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.065860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.065996 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.066617 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:22.066629 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:22.069490 (+ 2861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:22.071562 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:22.071563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:22.071564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:22.071918 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:22.072127 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:22.072127 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:22.072127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:22.072247 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:22.072593 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:22.072594 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:22.072684 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 23:16:22.073160 (+ 476us) client_negotiation.cc:240] Negotiation successful
0924 23:16:22.073517 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":355,"thread_start_us":69,"threads_started":1}
I20260924 23:16:22.073642 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.060604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:45945 (local address 127.1.4.194:43725)
0924 23:16:22.060781 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:22.060821 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:22.061387 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:22.062059 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:22.062061 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:22.062566 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:22.062879 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.062885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.063929 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.063930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.065350 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.065352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.065858 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.065858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.065995 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.066661 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:22.066666 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:22.069490 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:22.071529 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:22.071536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:22.071539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:22.071956 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:22.072153 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:22.072153 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:22.072154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:22.072242 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:22.072509 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:22.072515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:22.072683 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 23:16:22.073154 (+ 471us) client_negotiation.cc:240] Negotiation successful
0924 23:16:22.073429 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":211,"thread_start_us":79,"threads_started":1}
I20260924 23:16:22.073786 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.060604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.194:37735)
0924 23:16:22.060780 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:22.060821 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:22.061380 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:22.062299 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:22.062300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:22.062572 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:22.062879 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.062886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.063930 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.063931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.065349 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.065350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.065859 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.065860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.065996 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.066774 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:22.066779 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:22.069332 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:22.071688 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:22.071688 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:22.071689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:22.072297 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:22.072465 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:22.072465 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:22.072466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:22.072527 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:22.072795 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:22.072795 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:22.072831 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 23:16:22.073153 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 23:16:22.073362 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":155,"thread_start_us":79,"threads_started":1}
I20260924 23:16:22.074162 8247 heartbeater.cc:347] Connected to a master server at 127.1.4.252:45637
I20260924 23:16:22.074257 8247 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:22.074326 8248 heartbeater.cc:347] Connected to a master server at 127.1.4.254:42079
I20260924 23:16:22.074381 8248 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:22.074452 8247 heartbeater.cc:511] Master 127.1.4.252:45637 requested a full tablet report, sending...
I20260924 23:16:22.074465 8248 heartbeater.cc:511] Master 127.1.4.254:42079 requested a full tablet report, sending...
I20260924 23:16:22.074651 8249 heartbeater.cc:347] Connected to a master server at 127.1.4.253:45945
I20260924 23:16:22.074690 8249 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:22.074739 8249 heartbeater.cc:511] Master 127.1.4.253:45945 requested a full tablet report, sending...
I20260924 23:16:22.074889 7146 ts_manager.cc:194] Re-registered known tserver with Master: e85bb432f60642fb92fa4c620b6ac7a0 (127.1.4.194:37679)
I20260924 23:16:22.074951 7930 ts_manager.cc:194] Registered new tserver with Master: e85bb432f60642fb92fa4c620b6ac7a0 (127.1.4.194:37679)
I20260924 23:16:22.075027 7744 ts_manager.cc:194] Registered new tserver with Master: e85bb432f60642fb92fa4c620b6ac7a0 (127.1.4.194:37679)
I20260924 23:16:22.075356 7146 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:37735
I20260924 23:16:22.115115 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.112038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:45637 (local address 127.0.0.1:34924)
0924 23:16:22.112099 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:22.112109 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:22.112159 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:22.112403 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:22.112404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:22.112406 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:22.112464 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.112466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.113438 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.113438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.114154 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.114156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.114894 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.114896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.114945 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.114949 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:22.115017 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 23:16:22.115053 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 23:16:22.115139 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.112113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.1.4.252:45637)
0924 23:16:22.112154 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.112156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.112215 (+ 59us) server_negotiation.cc:407] Connection header received
0924 23:16:22.112266 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.112267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.112288 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.112376 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:22.112554 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.113351 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.114237 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.114814 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.114884 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.114958 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.115030 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.115069 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:16:22.115500 7940 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 23:16:22.115661 7969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3a137b4a02d44709e97e6af2f7dd626. Latest consensus state: current_term: 1 leader_uuid: "c3a137b4a02d44709e97e6af2f7dd626" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } } }
I20260924 23:16:22.115772 7969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [sys.catalog]: This master's current role is: LEARNER
W20260924 23:16:22.153115 8254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:22.153308 8254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:22.153331 8254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:22.154455 8254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:22.154496 8254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:22.154520 8254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:16:22.155992 8254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.195.keytab
--principal=oryx/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:36637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=43247
--tserver_master_addrs=127.1.4.254:42079,127.1.4.253:45945,127.1.4.252:45637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:22.156208 8254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:22.156421 8254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:22.161209 8260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:22.161221 8262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:22.161305 8254 server_base.cc:1034] running on GCE node
W20260924 23:16:22.161211 8259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:22.161558 8254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:22.161790 8254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:22.162941 8254 hybrid_clock.cc:648] HybridClock initialized: now 1790291782162924 us; error 31 us; skew 500 ppm
Sep 24 23:16:22 dist-test-slave-s1hw krb5kdc[7101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291782, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:22.165315 8254 init.cc:377] Logged in from keytab as oryx/127.1.4.195@KRBTEST.COM (short username oryx)
I20260924 23:16:22.165760 8254 webserver.cc:460] Webserver started at http://127.1.4.195:43247/ using document root <none> and password file <none>
I20260924 23:16:22.165975 8254 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:22.166030 8254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:22.167277 8254 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:22.167929 8269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:22.168082 8254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:16:22.168151 8254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "3eec19b8e01544c7a91a6ab99e738b73"
format_stamp: "Formatted at 2026-09-24 23:16:21 on dist-test-slave-s1hw"
server_key: "f2ce0de9d24849dd794a3b040307ebc5"
server_key_iv: "548ffc99845afca4d328e7b35b3c5a36"
server_key_version: "encryptionkey@0"
I20260924 23:16:22.168500 8254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:22.186913 8254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:22.188941 8254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:22.189067 8254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:22.189301 8254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:22.189592 8254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:22.189625 8254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:22.189646 8254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:22.189669 8254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:22.195240 8254 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:36637
I20260924 23:16:22.195288 8382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:36637 every 8 connection(s)
I20260924 23:16:22.195652 8254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
Sep 24 23:16:22 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291782, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM
Sep 24 23:16:22 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291782, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.252@KRBTEST.COM
Sep 24 23:16:22 dist-test-slave-s1hw krb5kdc[7101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291782, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.253@KRBTEST.COM
I20260924 23:16:22.205120 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 8254
I20260924 23:16:22.207475 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.196423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:33503 (local address 127.1.4.253:45945)
0924 23:16:22.196463 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.196465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.196853 (+ 388us) server_negotiation.cc:407] Connection header received
0924 23:16:22.197729 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.197730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.197744 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.197786 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:22.198678 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.199487 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.201516 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.201800 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.201859 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.204521 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:22.204522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:22.204523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:22.204531 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:22.205794 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.206262 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.206263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.206263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.206324 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.206635 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.206636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.206637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.206781 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:22.206828 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.207251 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.207351 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:16:22.207633 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.196747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:52273 (local address 127.1.4.252:45637)
0924 23:16:22.196794 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.196796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.196886 (+ 90us) server_negotiation.cc:407] Connection header received
0924 23:16:22.197944 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.197945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.197955 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.198000 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:22.199465 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.200163 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.200954 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.201248 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.201314 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.204246 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:22.204248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:22.204248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:22.204256 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:22.205668 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.206834 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.206835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.206836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.206896 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.207084 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.207084 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.207085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.207208 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:22.207266 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.207470 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.207546 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:16:22.207870 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.196369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:36595 (local address 127.1.4.254:42079)
0924 23:16:22.196412 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.196413 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.196660 (+ 247us) server_negotiation.cc:407] Connection header received
0924 23:16:22.197843 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.197845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.197854 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.197892 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:22.198652 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.199475 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.200712 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.200989 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.201045 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.203408 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:22.203410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:22.203410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:22.203419 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:22.205003 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.207054 (+ 2051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.207055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.207056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.207120 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.207316 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.207317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.207317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.207475 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:22.207513 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.207707 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.207786 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:16:22.208138 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.196670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:45637 (local address 127.1.4.195:52273)
0924 23:16:22.196822 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:22.196831 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:22.197617 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:22.198028 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:22.198029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:22.198262 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:22.198563 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.198569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.200251 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.200252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.200868 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.200870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.201322 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.201322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.201668 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.202141 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:22.202144 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:22.204141 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:22.205752 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:22.205760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:22.205763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:22.206506 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:22.206981 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:22.206981 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:22.206982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:22.207039 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:22.207290 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:22.207291 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:22.207331 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 23:16:22.207751 (+ 420us) client_negotiation.cc:240] Negotiation successful
0924 23:16:22.207889 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":83,"thread_start_us":66,"threads_started":1}
I20260924 23:16:22.208138 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.196385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.195:36595)
0924 23:16:22.196546 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:22.196583 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:22.197611 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:22.197913 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:22.197914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:22.198300 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:22.198563 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.198569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.199600 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.199601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.200637 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.200639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.201054 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.201055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.201180 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.201668 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:22.201676 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:22.203240 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:22.206571 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:22.206571 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:22.206572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:22.206747 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:22.207190 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:22.207191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:22.207191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:22.207239 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:22.207557 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:22.207558 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:22.207606 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 23:16:22.207958 (+ 352us) client_negotiation.cc:240] Negotiation successful
0924 23:16:22.208042 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":22,"thread_start_us":95,"threads_started":1}
I20260924 23:16:22.208586 8384 heartbeater.cc:347] Connected to a master server at 127.1.4.254:42079
I20260924 23:16:22.208693 8384 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:22.208786 8383 heartbeater.cc:347] Connected to a master server at 127.1.4.252:45637
I20260924 23:16:22.208837 8383 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:22.208915 8384 heartbeater.cc:511] Master 127.1.4.254:42079 requested a full tablet report, sending...
I20260924 23:16:22.208916 8383 heartbeater.cc:511] Master 127.1.4.252:45637 requested a full tablet report, sending...
I20260924 23:16:22.208928 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.196394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:45945 (local address 127.1.4.195:33503)
0924 23:16:22.196794 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:22.196803 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:22.197621 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:22.197872 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:22.197876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:22.198302 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:22.198563 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.198569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.200709 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.200709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.201451 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.201453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.201863 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.201864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.201917 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.202297 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:22.202300 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:22.204403 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:22.205878 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:22.205879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:22.205879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:22.206187 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:22.206403 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:22.206404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:22.206404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:22.206549 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:22.206880 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:22.206887 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:22.207089 (+ 202us) client_negotiation.cc:769] Sending connection context
0924 23:16:22.207631 (+ 542us) client_negotiation.cc:240] Negotiation successful
0924 23:16:22.207836 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":20,"thread_start_us":87,"threads_started":1}
I20260924 23:16:22.209329 7146 ts_manager.cc:194] Re-registered known tserver with Master: 3eec19b8e01544c7a91a6ab99e738b73 (127.1.4.195:36637)
I20260924 23:16:22.209362 7930 ts_manager.cc:194] Registered new tserver with Master: 3eec19b8e01544c7a91a6ab99e738b73 (127.1.4.195:36637)
I20260924 23:16:22.209448 8385 heartbeater.cc:347] Connected to a master server at 127.1.4.253:45945
I20260924 23:16:22.209510 8385 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:22.209570 8385 heartbeater.cc:511] Master 127.1.4.253:45945 requested a full tablet report, sending...
I20260924 23:16:22.209797 7146 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:36595
I20260924 23:16:22.209838 7744 ts_manager.cc:194] Registered new tserver with Master: 3eec19b8e01544c7a91a6ab99e738b73 (127.1.4.195:36637)
I20260924 23:16:22.216881 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.206954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45992 (local address 127.1.4.254:42079)
0924 23:16:22.207434 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.207436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.207443 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:16:22.207463 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.207464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.207476 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.208102 (+ 626us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:22.210394 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.211229 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.211966 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.212187 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.212521 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.214184 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:22.214185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:22.214186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:22.214192 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:22.215695 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.216064 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.216065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.216066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.216121 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.216349 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.216350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.216350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.216465 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:22.216553 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.216689 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.216778 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":63,"threads_started":1}
I20260924 23:16:22.217645 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.206666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.1.4.253:45945)
0924 23:16:22.207561 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.207563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.210699 (+ 3136us) server_negotiation.cc:407] Connection header received
0924 23:16:22.210742 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.210743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.210753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.210804 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:22.211419 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.212104 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.213060 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.213339 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.213400 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.214819 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:22.214821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:22.214821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:22.214828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:22.216291 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.216871 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.216872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.216873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.216932 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.217125 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.217126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.217127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.217260 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:22.217364 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.217475 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.217549 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":58,"threads_started":1}
I20260924 23:16:22.217891 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.206943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.1.4.252:45637)
0924 23:16:22.207708 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.207710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.210515 (+ 2805us) server_negotiation.cc:407] Connection header received
0924 23:16:22.210558 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.210559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.210569 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.210629 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:22.211297 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.212064 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.213023 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.213240 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.213301 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.214810 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:22.214812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:22.214812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:22.214819 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:22.216385 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.216700 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.216701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.216702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.216760 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.217319 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.217320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.217321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.217452 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:22.217530 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.217748 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.217820 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":64,"threads_started":1}
I20260924 23:16:22.218544 7146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:16:22.220374 7146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:22.221119 7706 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3a137b4a02d44709e97e6af2f7dd626" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: NON_VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: true } } }
I20260924 23:16:22.221215 7706 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:22.222541 8401 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626: attempting to promote NON_VOTER 5c7fbf9789ba4b5fb7c52c61869193ea to VOTER
I20260924 23:16:22.222721 8401 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } }
I20260924 23:16:22.223304 7940 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 1 LEARNER]: Refusing update from remote peer c3a137b4a02d44709e97e6af2f7dd626: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260924 23:16:22.223340 7754 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 1 FOLLOWER]: Refusing update from remote peer c3a137b4a02d44709e97e6af2f7dd626: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260924 23:16:22.223521 8401 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 23:16:22.223628 8402 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 23:16:22.224222 8401 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5c7fbf9789ba4b5fb7c52c61869193ea (127.1.4.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } } }
I20260924 23:16:22.224392 7754 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5c7fbf9789ba4b5fb7c52c61869193ea (127.1.4.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } } }
I20260924 23:16:22.224378 7940 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5c7fbf9789ba4b5fb7c52c61869193ea (127.1.4.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } } }
I20260924 23:16:22.225267 8402 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c3a137b4a02d44709e97e6af2f7dd626" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } } }
I20260924 23:16:22.225345 8402 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:22.225312 7969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3a137b4a02d44709e97e6af2f7dd626" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } } }
I20260924 23:16:22.225382 7969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:16:22.225600 8411 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3a137b4a02d44709e97e6af2f7dd626" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } } }
I20260924 23:16:22.225678 8411 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:16:22.225994 8401 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3a137b4a02d44709e97e6af2f7dd626" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } } }
I20260924 23:16:22.226083 8403 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3a137b4a02d44709e97e6af2f7dd626" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } } }
I20260924 23:16:22.226279 8401 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:22.226294 8403 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:22.230358 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.225967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:34339 (local address 127.0.0.1:47220)
0924 23:16:22.226303 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:22.226315 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:22.226355 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:22.226572 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:22.226573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:22.226575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:22.226621 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.226623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.228193 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.228194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.229260 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.229262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.230114 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.230115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.230182 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.230186 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:22.230261 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 23:16:22.230287 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":5}
I20260924 23:16:22.230367 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.226056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:36637 (local address 127.0.0.1:41192)
0924 23:16:22.226313 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:22.226324 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:22.226372 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:22.227342 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:22.227343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:22.227345 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:22.227410 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.227412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.228472 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.228473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.229174 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.229176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.230115 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.230115 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.230182 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.230186 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:22.230239 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 23:16:22.230278 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260924 23:16:22.230374 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.226504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.1.4.195:36637)
0924 23:16:22.226972 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.226977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.226987 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:16:22.227179 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.227182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.227196 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.227344 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:22.227475 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.228403 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.229249 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.230025 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.230109 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.230188 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.230261 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.230299 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":73,"threads_started":1}
I20260924 23:16:22.231055 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.226053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.1.4.193:34339)
0924 23:16:22.226412 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.226417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.226428 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:16:22.226467 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.226469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.226485 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.226562 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:22.226678 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.228123 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.229327 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.230035 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.230109 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.230186 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.230934 (+ 748us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.230978 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":74,"threads_started":1}
I20260924 23:16:22.231307 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.226585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.1.4.194:37679)
0924 23:16:22.226734 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.226739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.226849 (+ 110us) server_negotiation.cc:407] Connection header received
0924 23:16:22.226992 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.226994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.227007 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.227084 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:22.227205 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.227983 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.230222 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.230973 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.231045 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.231121 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.231195 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.231233 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260924 23:16:22.231283 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.226197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:37679 (local address 127.0.0.1:49282)
0924 23:16:22.226785 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:22.226796 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:22.226848 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:22.227089 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:22.227090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:22.227092 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:22.227133 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.227135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.228053 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.228054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.229260 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.229262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.231072 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.231073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.231129 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.231133 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:22.231182 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 23:16:22.231219 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":30,"thread_start_us":71,"threads_started":1}
I20260924 23:16:22.231341 8317 tablet_service.cc:1467] Processing CreateTablet for tablet 77998ddf498d47a4902e90b9fe6e9d7a (DEFAULT_TABLE table=test-table [id=4c24d0cb64d64dd7a3abbdb3ef0994ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:22.231711 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77998ddf498d47a4902e90b9fe6e9d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:22.231690 8045 tablet_service.cc:1467] Processing CreateTablet for tablet 77998ddf498d47a4902e90b9fe6e9d7a (DEFAULT_TABLE table=test-table [id=4c24d0cb64d64dd7a3abbdb3ef0994ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:22.231962 8045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77998ddf498d47a4902e90b9fe6e9d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:22.232092 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 77998ddf498d47a4902e90b9fe6e9d7a (DEFAULT_TABLE table=test-table [id=4c24d0cb64d64dd7a3abbdb3ef0994ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:22.232379 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77998ddf498d47a4902e90b9fe6e9d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:22.234534 8418 tablet_bootstrap.cc:492] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c: Bootstrap starting.
I20260924 23:16:22.235045 8418 tablet_bootstrap.cc:654] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:22.235348 8419 tablet_bootstrap.cc:492] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73: Bootstrap starting.
I20260924 23:16:22.235977 8420 tablet_bootstrap.cc:492] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0: Bootstrap starting.
I20260924 23:16:22.236061 8419 tablet_bootstrap.cc:654] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:22.236200 8418 log.cc:824] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:22.236334 8419 log.cc:824] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:22.236547 8420 tablet_bootstrap.cc:654] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:22.236868 8420 log.cc:824] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:22.236999 8418 tablet_bootstrap.cc:492] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c: No bootstrap required, opened a new log
I20260924 23:16:22.237056 8418 ts_tablet_manager.cc:1397] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 23:16:22.237566 8420 tablet_bootstrap.cc:492] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0: No bootstrap required, opened a new log
I20260924 23:16:22.237619 8420 ts_tablet_manager.cc:1397] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:22.238446 8418 raft_consensus.cc:348] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:22.238571 8418 raft_consensus.cc:374] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:22.238591 8418 raft_consensus.cc:729] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ab7c555b634abda9f23607f5ee397c, State: Initialized, Role: FOLLOWER
I20260924 23:16:22.238689 8418 consensus_queue.cc:260] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:22.238920 8418 ts_tablet_manager.cc:1428] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:16:22.238945 8112 heartbeater.cc:503] Master 127.1.4.254:42079 was elected leader, sending a full tablet report...
I20260924 23:16:22.239773 8419 tablet_bootstrap.cc:492] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73: No bootstrap required, opened a new log
I20260924 23:16:22.239682 8420 raft_consensus.cc:348] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:22.239822 8420 raft_consensus.cc:374] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:22.239857 8420 raft_consensus.cc:729] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85bb432f60642fb92fa4c620b6ac7a0, State: Initialized, Role: FOLLOWER
I20260924 23:16:22.239960 8419 ts_tablet_manager.cc:1397] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 23:16:22.239974 8420 consensus_queue.cc:260] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:22.240216 8420 ts_tablet_manager.cc:1428] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:16:22.240413 8248 heartbeater.cc:503] Master 127.1.4.254:42079 was elected leader, sending a full tablet report...
I20260924 23:16:22.242291 8419 raft_consensus.cc:348] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:22.242424 8419 raft_consensus.cc:374] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:22.242444 8419 raft_consensus.cc:729] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eec19b8e01544c7a91a6ab99e738b73, State: Initialized, Role: FOLLOWER
I20260924 23:16:22.242547 8419 consensus_queue.cc:260] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:22.242756 8419 ts_tablet_manager.cc:1428] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:16:22.242796 8384 heartbeater.cc:503] Master 127.1.4.254:42079 was elected leader, sending a full tablet report...
W20260924 23:16:22.310617 8250 tablet.cc:2369] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:16:22.422842 8114 tablet.cc:2369] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:16:22.446432 8386 tablet.cc:2369] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:16:22.586072 8424 raft_consensus.cc:482] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:22.586198 8424 raft_consensus.cc:504] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:22.586592 8424 leader_election.cc:288] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eec19b8e01544c7a91a6ab99e738b73 (127.1.4.195:36637), e85bb432f60642fb92fa4c620b6ac7a0 (127.1.4.194:37679)
I20260924 23:16:22.589437 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.587074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:55201 (local address 127.1.4.194:37679)
0924 23:16:22.587116 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.587118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.587129 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:16:22.587203 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.587203 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.587214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.587260 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:22.587358 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.587840 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.588816 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.589160 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.589216 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.589226 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.589335 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.589368 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:16:22.589452 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.586839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:59865 (local address 127.1.4.195:36637)
0924 23:16:22.586893 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.586896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.587046 (+ 150us) server_negotiation.cc:407] Connection header received
0924 23:16:22.587107 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.587108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.587121 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.587168 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:22.587285 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.587834 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.588822 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.589159 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.589216 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.589226 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.589346 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.589383 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 23:16:22.589457 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.586972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:37679 (local address 127.1.4.193:55201)
0924 23:16:22.587079 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:22.587088 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:22.587143 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:22.587261 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:22.587262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:22.587263 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:22.587309 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.587310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.587909 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.587910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.588736 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.588738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.589226 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.589227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.589280 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.589283 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:22.589327 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:16:22.589367 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":25,"thread_start_us":38,"threads_started":1}
I20260924 23:16:22.589462 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.586792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:36637 (local address 127.1.4.193:59865)
0924 23:16:22.586916 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:22.586927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:22.587052 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:22.587176 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:22.587177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:22.587179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:22.587231 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.587233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.587906 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.587907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.588739 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:22.588741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.589222 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.589223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:22.589286 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.589290 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:22.589333 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 23:16:22.589373 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":28,"thread_start_us":73,"threads_started":1}
I20260924 23:16:22.589831 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77998ddf498d47a4902e90b9fe6e9d7a" candidate_uuid: "08ab7c555b634abda9f23607f5ee397c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" is_pre_election: true
I20260924 23:16:22.589975 8201 raft_consensus.cc:2454] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08ab7c555b634abda9f23607f5ee397c in term 0.
I20260924 23:16:22.590044 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77998ddf498d47a4902e90b9fe6e9d7a" candidate_uuid: "08ab7c555b634abda9f23607f5ee397c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eec19b8e01544c7a91a6ab99e738b73" is_pre_election: true
I20260924 23:16:22.590163 8337 raft_consensus.cc:2454] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08ab7c555b634abda9f23607f5ee397c in term 0.
I20260924 23:16:22.590174 8000 leader_election.cc:302] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ab7c555b634abda9f23607f5ee397c, e85bb432f60642fb92fa4c620b6ac7a0; no voters:
I20260924 23:16:22.590296 8424 raft_consensus.cc:2781] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:22.590360 8424 raft_consensus.cc:482] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:22.590385 8424 raft_consensus.cc:3037] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:22.591004 8424 raft_consensus.cc:504] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:22.591111 8424 leader_election.cc:288] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [CANDIDATE]: Term 1 election: Requested vote from peers 3eec19b8e01544c7a91a6ab99e738b73 (127.1.4.195:36637), e85bb432f60642fb92fa4c620b6ac7a0 (127.1.4.194:37679)
I20260924 23:16:22.591243 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77998ddf498d47a4902e90b9fe6e9d7a" candidate_uuid: "08ab7c555b634abda9f23607f5ee397c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eec19b8e01544c7a91a6ab99e738b73"
I20260924 23:16:22.591275 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77998ddf498d47a4902e90b9fe6e9d7a" candidate_uuid: "08ab7c555b634abda9f23607f5ee397c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85bb432f60642fb92fa4c620b6ac7a0"
I20260924 23:16:22.591311 8337 raft_consensus.cc:3037] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:22.591338 8201 raft_consensus.cc:3037] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:22.592005 8337 raft_consensus.cc:2454] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08ab7c555b634abda9f23607f5ee397c in term 1.
I20260924 23:16:22.592010 8201 raft_consensus.cc:2454] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08ab7c555b634abda9f23607f5ee397c in term 1.
I20260924 23:16:22.592162 8000 leader_election.cc:302] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ab7c555b634abda9f23607f5ee397c, 3eec19b8e01544c7a91a6ab99e738b73; no voters:
I20260924 23:16:22.592229 8424 raft_consensus.cc:2781] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:22.592337 8424 raft_consensus.cc:686] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [term 1 LEADER]: Becoming Leader. State: Replica: 08ab7c555b634abda9f23607f5ee397c, State: Running, Role: LEADER
I20260924 23:16:22.592417 8424 consensus_queue.cc:237] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:22.593113 7145 catalog_manager.cc:5495] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08ab7c555b634abda9f23607f5ee397c (127.1.4.193). New cstate: current_term: 1 leader_uuid: "08ab7c555b634abda9f23607f5ee397c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } health_report { overall_health: UNKNOWN } } }
I20260924 23:16:22.668349 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.665747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.1.4.193:34339)
0924 23:16:22.665799 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.665801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.665809 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:22.665845 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.665846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.665864 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.665922 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:22.666146 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.666837 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.667636 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.667841 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.667898 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.668042 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:22.668159 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:22.668233 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.668250 (+ 17us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.668271 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:16:22.670083 8337 raft_consensus.cc:1261] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73 [term 1 FOLLOWER]: Refusing update from remote peer 08ab7c555b634abda9f23607f5ee397c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:22.670094 8201 raft_consensus.cc:1261] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0 [term 1 FOLLOWER]: Refusing update from remote peer 08ab7c555b634abda9f23607f5ee397c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:22.670413 8429 consensus_queue.cc:1035] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:22.670572 8424 consensus_queue.cc:1035] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:22.672834 8432 mvcc.cc:205] Tried to move back new op lower bound from 7333035141814255616 to 7333035141499064320. Current Snapshot: MvccSnapshot[applied={T|T < 7333035141814255616}]
I20260924 23:16:22.674167 8433 mvcc.cc:205] Tried to move back new op lower bound from 7333035141814255616 to 7333035141499064320. Current Snapshot: MvccSnapshot[applied={T|T < 7333035141814255616}]
I20260924 23:16:22.675905 7145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:22.675987 7145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:22.677217 7145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3a137b4a02d44709e97e6af2f7dd626: Sending DeleteTablet for 3 replicas of tablet 77998ddf498d47a4902e90b9fe6e9d7a
I20260924 23:16:22.677557 8045 tablet_service.cc:1514] Processing DeleteTablet for tablet 77998ddf498d47a4902e90b9fe6e9d7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:22 UTC) from {username='oryx', principal='oryx/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:47220
I20260924 23:16:22.677552 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 77998ddf498d47a4902e90b9fe6e9d7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:22 UTC) from {username='oryx', principal='oryx/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:49282
I20260924 23:16:22.677632 8317 tablet_service.cc:1514] Processing DeleteTablet for tablet 77998ddf498d47a4902e90b9fe6e9d7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:22 UTC) from {username='oryx', principal='oryx/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:41192
I20260924 23:16:22.677831 8443 tablet_replica.cc:330] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73: stopping tablet replica
I20260924 23:16:22.677928 8443 raft_consensus.cc:2229] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:22.678021 8443 raft_consensus.cc:2258] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:22.678125 8442 tablet_replica.cc:330] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0: stopping tablet replica
I20260924 23:16:22.678261 8442 raft_consensus.cc:2229] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:22.678407 8442 raft_consensus.cc:2258] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:22.678462 8443 ts_tablet_manager.cc:1905] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:22.678925 8442 ts_tablet_manager.cc:1905] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:22.679136 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 7114
I20260924 23:16:22.680205 8443 ts_tablet_manager.cc:1918] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:22.680279 8443 log.cc:1196] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/77998ddf498d47a4902e90b9fe6e9d7a
I20260924 23:16:22.680557 8443 ts_tablet_manager.cc:1939] T 77998ddf498d47a4902e90b9fe6e9d7a P 3eec19b8e01544c7a91a6ab99e738b73: Deleting consensus metadata
I20260924 23:16:22.680907 8442 ts_tablet_manager.cc:1918] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:22.681023 8442 log.cc:1196] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/77998ddf498d47a4902e90b9fe6e9d7a
I20260924 23:16:22.681309 8442 ts_tablet_manager.cc:1939] T 77998ddf498d47a4902e90b9fe6e9d7a P e85bb432f60642fb92fa4c620b6ac7a0: Deleting consensus metadata
W20260924 23:16:22.684712 8136 connection.cc:535] server connection from 127.0.0.1:49282 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 23:16:22.684742 8272 connection.cc:535] server connection from 127.0.0.1:41192 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 23:16:22.684948 8441 tablet_replica.cc:330] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c: stopping tablet replica
I20260924 23:16:22.685050 8441 raft_consensus.cc:2229] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:16:22.685150 8441 raft_consensus.cc:2258] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:22.685578 8441 ts_tablet_manager.cc:1905] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:22.687152 8441 ts_tablet_manager.cc:1918] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:22.687223 8441 log.cc:1196] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-0/wal/wals/77998ddf498d47a4902e90b9fe6e9d7a
I20260924 23:16:22.687511 8441 ts_tablet_manager.cc:1939] T 77998ddf498d47a4902e90b9fe6e9d7a P 08ab7c555b634abda9f23607f5ee397c: Deleting consensus metadata
W20260924 23:16:22.687834 8001 connection.cc:441] server connection from 127.0.0.1:47220 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47220 (request call id 1) could send its response
I20260924 23:16:22.694131 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.686581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.1.4.253:45945)
0924 23:16:22.686628 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.686629 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.686765 (+ 136us) server_negotiation.cc:407] Connection header received
0924 23:16:22.686861 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.686862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.686872 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.686903 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:22.687807 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.688581 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.689694 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.690006 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.690080 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.691488 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:22.691489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:22.691490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:22.691497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:22.692973 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.693368 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.693369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.693370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.693425 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.693589 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.693590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.693590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.693696 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:22.693795 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.693883 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.693968 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:16:22.694275 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:22.686805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.1.4.252:45637)
0924 23:16:22.686847 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:22.686848 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:22.687085 (+ 237us) server_negotiation.cc:407] Connection header received
0924 23:16:22.687171 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:22.687172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:22.687182 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:22.687210 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:22.687867 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.688588 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.690137 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:22.690439 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:22.690512 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:22.691788 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:22.691789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:22.691790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:22.691797 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:22.693066 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.693489 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.693490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.693490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.693546 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:22.693739 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:22.693740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:22.693740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:22.693858 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:22.693921 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:22.694108 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 23:16:22.694182 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:16:23.675196 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:23.674925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.193:58201)
0924 23:16:23.675064 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:23.675144 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260924 23:16:23.675196 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:23.674925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.195:43283)
0924 23:16:23.675075 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:23.675145 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
W20260924 23:16:23.675676 8112 heartbeater.cc:644] Failed to heartbeat to 127.1.4.254:42079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
W20260924 23:16:23.675671 8384 heartbeater.cc:644] Failed to heartbeat to 127.1.4.254:42079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
I20260924 23:16:23.676110 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:23.675850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.194:35583)
0924 23:16:23.675950 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:23.676065 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":44,"threads_started":1}
W20260924 23:16:23.676342 8248 heartbeater.cc:644] Failed to heartbeat to 127.1.4.254:42079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
I20260924 23:16:24.177687 8454 raft_consensus.cc:482] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3a137b4a02d44709e97e6af2f7dd626)
I20260924 23:16:24.177811 8454 raft_consensus.cc:504] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } }
I20260924 23:16:24.178056 8454 leader_election.cc:288] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3a137b4a02d44709e97e6af2f7dd626 (127.1.4.254:42079), 5c7fbf9789ba4b5fb7c52c61869193ea (127.1.4.252:45637)
I20260924 23:16:24.178354 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.178170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.0.0.1:46486)
0924 23:16:24.178265 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.178317 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":43,"threads_started":1}
W20260924 23:16:24.178535 7729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
I20260924 23:16:24.178589 8457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c3a137b4a02d44709e97e6af2f7dd626)
I20260924 23:16:24.178700 8457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } }
I20260924 23:16:24.178925 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.178837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.0.0.1:46488)
0924 23:16:24.178878 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.178900 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 23:16:24.178937 8457 leader_election.cc:288] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3a137b4a02d44709e97e6af2f7dd626 (127.1.4.254:42079), b69cbd28a4394c14aaff8bb3901f5472 (127.1.4.253:45945)
W20260924 23:16:24.179066 7729 leader_election.cc:334] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c3a137b4a02d44709e97e6af2f7dd626 (127.1.4.254:42079): Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
I20260924 23:16:24.179294 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.179086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.0.0.1:46504)
0924 23:16:24.179213 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.179263 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":52,"threads_started":1}
W20260924 23:16:24.179438 7915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
I20260924 23:16:24.179816 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.179730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.0.0.1:46508)
0924 23:16:24.179762 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.179790 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 23:16:24.179929 7915 leader_election.cc:334] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c3a137b4a02d44709e97e6af2f7dd626 (127.1.4.254:42079): Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
I20260924 23:16:24.181495 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.178517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34944 (local address 127.1.4.252:45637)
0924 23:16:24.178619 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:24.178621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:24.178629 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:24.178655 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:24.178656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:24.178666 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:24.178887 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:24.178917 (+ 30us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.179660 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.180837 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.181208 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.181263 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:24.181272 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:24.181391 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 23:16:24.181434 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":44,"threads_started":1}
I20260924 23:16:24.181509 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.178289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:45637 (local address 127.0.0.1:34944)
0924 23:16:24.178384 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.178393 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:24.178444 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:24.178765 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:24.178766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:24.178768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:24.178830 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:24.178832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.179722 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.179723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:24.180762 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:24.180765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.181279 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.181280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:24.181338 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:24.181343 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:16:24.181389 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 23:16:24.181450 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":17,"thread_start_us":30,"threads_started":1}
I20260924 23:16:24.181787 7940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b69cbd28a4394c14aaff8bb3901f5472" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" is_pre_election: true
I20260924 23:16:24.181872 7940 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b69cbd28a4394c14aaff8bb3901f5472 in term 1.
I20260924 23:16:24.182013 7729 leader_election.cc:302] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c7fbf9789ba4b5fb7c52c61869193ea, b69cbd28a4394c14aaff8bb3901f5472; no voters: c3a137b4a02d44709e97e6af2f7dd626
I20260924 23:16:24.182111 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.179179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.1.4.253:45945)
0924 23:16:24.179295 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:24.179296 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:24.179303 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:16:24.179321 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:24.179322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:24.179330 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:24.179440 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:24.179651 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.180255 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.181329 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.181823 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.181877 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:24.181888 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:24.182022 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 23:16:24.182054 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260924 23:16:24.182121 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.179033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:45945 (local address 127.0.0.1:45526)
0924 23:16:24.179180 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.179189 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:24.179238 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:24.179466 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:24.179466 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:24.179468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:24.179562 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:24.179565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.180327 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.180328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:24.181259 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:24.181262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.181888 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.181889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:24.181958 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:24.181961 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:24.182016 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 23:16:24.182065 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260924 23:16:24.182137 8454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 23:16:24.182163 8454 raft_consensus.cc:482] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c3a137b4a02d44709e97e6af2f7dd626)
I20260924 23:16:24.182180 8454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 1 FOLLOWER]: Advancing to term 2
I20260924 23:16:24.182299 7754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b69cbd28a4394c14aaff8bb3901f5472" is_pre_election: true
I20260924 23:16:24.182869 8454 raft_consensus.cc:504] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } }
I20260924 23:16:24.182979 8454 leader_election.cc:288] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [CANDIDATE]: Term 2 election: Requested vote from peers c3a137b4a02d44709e97e6af2f7dd626 (127.1.4.254:42079), 5c7fbf9789ba4b5fb7c52c61869193ea (127.1.4.252:45637)
I20260924 23:16:24.182986 7754 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c7fbf9789ba4b5fb7c52c61869193ea in current term 2: Already voted for candidate b69cbd28a4394c14aaff8bb3901f5472 in this term.
I20260924 23:16:24.183136 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.183058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.0.0.1:46520)
0924 23:16:24.183088 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.183117 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 23:16:24.183187 7917 leader_election.cc:302] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c7fbf9789ba4b5fb7c52c61869193ea; no voters: b69cbd28a4394c14aaff8bb3901f5472, c3a137b4a02d44709e97e6af2f7dd626
I20260924 23:16:24.183317 8457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 1 FOLLOWER]: Advancing to term 2
I20260924 23:16:24.183415 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.183310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.0.0.1:46536)
0924 23:16:24.183353 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.183398 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 23:16:24.183471 7729 leader_election.cc:334] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c3a137b4a02d44709e97e6af2f7dd626 (127.1.4.254:42079): Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
I20260924 23:16:24.183961 8457 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:16:24.184103 7940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b69cbd28a4394c14aaff8bb3901f5472" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea"
I20260924 23:16:24.184629 7940 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b69cbd28a4394c14aaff8bb3901f5472 in term 2.
I20260924 23:16:24.184764 7729 leader_election.cc:302] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c7fbf9789ba4b5fb7c52c61869193ea, b69cbd28a4394c14aaff8bb3901f5472; no voters: c3a137b4a02d44709e97e6af2f7dd626
I20260924 23:16:24.184850 8454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 23:16:24.184990 8454 raft_consensus.cc:686] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [term 2 LEADER]: Becoming Leader. State: Replica: b69cbd28a4394c14aaff8bb3901f5472, State: Running, Role: LEADER
I20260924 23:16:24.185061 8454 consensus_queue.cc:237] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } }
I20260924 23:16:24.185384 8464 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b69cbd28a4394c14aaff8bb3901f5472. Latest consensus state: current_term: 2 leader_uuid: "b69cbd28a4394c14aaff8bb3901f5472" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } } }
I20260924 23:16:24.185503 8464 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:24.185667 8466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:24.186103 8466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:24.186177 8466 catalog_manager.cc:1259] Loaded cluster ID: 79e249b6f5f4479fa2bb7f5060558ed9
I20260924 23:16:24.186208 8466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:16:24.186364 8466 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:24.186437 8466 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: Loaded TSK: 0
I20260924 23:16:24.186589 8466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:16:24.230413 7743 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:16:24.232088 7743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:24.233052 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.232951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.0.0.1:46560)
0924 23:16:24.232987 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.233021 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 23:16:24.233031 7940 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [term 2 FOLLOWER]: Refusing update from remote peer b69cbd28a4394c14aaff8bb3901f5472: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 23:16:24.233263 8464 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 23:16:24.233381 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.233291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.0.0.1:46572)
0924 23:16:24.233334 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.233361 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 23:16:24.233479 7729 consensus_peers.cc:487] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 -> Peer c3a137b4a02d44709e97e6af2f7dd626 (127.1.4.254:42079): Couldn't send request to peer c3a137b4a02d44709e97e6af2f7dd626. Status: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 23:16:24.233858 8457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader b69cbd28a4394c14aaff8bb3901f5472. Latest consensus state: current_term: 2 leader_uuid: "b69cbd28a4394c14aaff8bb3901f5472" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } } }
I20260924 23:16:24.233944 8457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:16:24.234325 8464 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b69cbd28a4394c14aaff8bb3901f5472" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } } }
I20260924 23:16:24.234405 8464 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:24.234504 8457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b69cbd28a4394c14aaff8bb3901f5472" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c3a137b4a02d44709e97e6af2f7dd626" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42079 } } peers { permanent_uuid: "b69cbd28a4394c14aaff8bb3901f5472" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 45945 } attrs { promote: false } } peers { permanent_uuid: "5c7fbf9789ba4b5fb7c52c61869193ea" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 45637 } attrs { promote: false } } }
I20260924 23:16:24.234593 8457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7fbf9789ba4b5fb7c52c61869193ea [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:16:24.239153 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.236063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:37679 (local address 127.0.0.1:49288)
0924 23:16:24.236151 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.236160 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:24.236201 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:24.236660 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:24.236661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:24.236663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:24.236695 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:24.236696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.237522 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.237523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:24.238209 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:24.238211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.238935 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.238936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:24.239010 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:24.239014 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:24.239041 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 23:16:24.239081 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":56,"threads_started":1}
I20260924 23:16:24.239264 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.236002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:34339 (local address 127.0.0.1:47234)
0924 23:16:24.236157 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.236166 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:24.236205 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:24.236397 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:24.236398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:24.236400 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:24.236445 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:24.236446 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.237305 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.237306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:24.238386 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:24.238388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.239073 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.239074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:24.239123 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:24.239127 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:24.239174 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 23:16:24.239212 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260924 23:16:24.239285 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.236050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.1.4.193:34339)
0924 23:16:24.236287 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:24.236288 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:24.236295 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:16:24.236316 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:24.236317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:24.236326 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:24.236382 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:24.236519 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.237233 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.238531 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.238977 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.239075 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:24.239087 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:24.239177 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 23:16:24.239216 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":64,"threads_started":1}
I20260924 23:16:24.239298 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.235989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.1.4.195:36637)
0924 23:16:24.236094 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:24.236096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:24.236103 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:16:24.236127 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:24.236127 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:24.236137 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:24.236380 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:24.236416 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.237092 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.238372 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.238934 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.238994 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:24.239009 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:24.239182 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 23:16:24.239228 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":62,"threads_started":1}
I20260924 23:16:24.239336 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.235910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:36637 (local address 127.0.0.1:41202)
0924 23:16:24.235938 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.235958 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:24.236002 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:24.236220 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:24.236222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:24.236223 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:24.236289 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:24.236291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.237157 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.237157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:24.238293 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:24.238295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.239003 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.239003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:24.239061 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:24.239065 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:24.239110 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 23:16:24.239149 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15,"mutex_wait_us":29}
I20260924 23:16:24.239573 8317 tablet_service.cc:1467] Processing CreateTablet for tablet 84d43c19453f4a40a4a81e3f5f4c6498 (DEFAULT_TABLE table=test-table [id=9be77a64628a4148809c6cf3e06e53c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:24.239687 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.236308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.1.4.194:37679)
0924 23:16:24.236542 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:24.236544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:24.236551 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:16:24.236573 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:24.236574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:24.236584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:24.236746 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:24.236790 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.237436 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.238283 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.238844 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.239542 (+ 698us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:24.239560 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:24.239578 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 23:16:24.239619 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":57,"threads_started":1}
I20260924 23:16:24.239701 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d43c19453f4a40a4a81e3f5f4c6498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:24.240734 8476 tablet_bootstrap.cc:492] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73: Bootstrap starting.
I20260924 23:16:24.240756 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 84d43c19453f4a40a4a81e3f5f4c6498 (DEFAULT_TABLE table=test-table [id=9be77a64628a4148809c6cf3e06e53c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:24.240862 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d43c19453f4a40a4a81e3f5f4c6498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:24.241115 8476 tablet_bootstrap.cc:654] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:24.241619 8476 tablet_bootstrap.cc:492] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73: No bootstrap required, opened a new log
I20260924 23:16:24.241659 8476 ts_tablet_manager.cc:1397] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:24.241793 8476 raft_consensus.cc:348] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:24.241850 8476 raft_consensus.cc:374] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:24.241868 8476 raft_consensus.cc:729] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eec19b8e01544c7a91a6ab99e738b73, State: Initialized, Role: FOLLOWER
I20260924 23:16:24.241909 8476 consensus_queue.cc:260] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:24.242054 8478 tablet_bootstrap.cc:492] T 84d43c19453f4a40a4a81e3f5f4c6498 P e85bb432f60642fb92fa4c620b6ac7a0: Bootstrap starting.
I20260924 23:16:24.242095 8045 tablet_service.cc:1467] Processing CreateTablet for tablet 84d43c19453f4a40a4a81e3f5f4c6498 (DEFAULT_TABLE table=test-table [id=9be77a64628a4148809c6cf3e06e53c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:24.242201 8045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d43c19453f4a40a4a81e3f5f4c6498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:24.242360 8478 tablet_bootstrap.cc:654] T 84d43c19453f4a40a4a81e3f5f4c6498 P e85bb432f60642fb92fa4c620b6ac7a0: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:24.242821 8478 tablet_bootstrap.cc:492] T 84d43c19453f4a40a4a81e3f5f4c6498 P e85bb432f60642fb92fa4c620b6ac7a0: No bootstrap required, opened a new log
I20260924 23:16:24.242861 8478 ts_tablet_manager.cc:1397] T 84d43c19453f4a40a4a81e3f5f4c6498 P e85bb432f60642fb92fa4c620b6ac7a0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:24.242975 8478 raft_consensus.cc:348] T 84d43c19453f4a40a4a81e3f5f4c6498 P e85bb432f60642fb92fa4c620b6ac7a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:24.243032 8478 raft_consensus.cc:374] T 84d43c19453f4a40a4a81e3f5f4c6498 P e85bb432f60642fb92fa4c620b6ac7a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:24.243045 8478 raft_consensus.cc:729] T 84d43c19453f4a40a4a81e3f5f4c6498 P e85bb432f60642fb92fa4c620b6ac7a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85bb432f60642fb92fa4c620b6ac7a0, State: Initialized, Role: FOLLOWER
I20260924 23:16:24.243086 8478 consensus_queue.cc:260] T 84d43c19453f4a40a4a81e3f5f4c6498 P e85bb432f60642fb92fa4c620b6ac7a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:24.243257 8478 ts_tablet_manager.cc:1428] T 84d43c19453f4a40a4a81e3f5f4c6498 P e85bb432f60642fb92fa4c620b6ac7a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:24.243273 8481 tablet_bootstrap.cc:492] T 84d43c19453f4a40a4a81e3f5f4c6498 P 08ab7c555b634abda9f23607f5ee397c: Bootstrap starting.
I20260924 23:16:24.243789 8481 tablet_bootstrap.cc:654] T 84d43c19453f4a40a4a81e3f5f4c6498 P 08ab7c555b634abda9f23607f5ee397c: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:24.243861 8249 heartbeater.cc:503] Master 127.1.4.253:45945 was elected leader, sending a full tablet report...
I20260924 23:16:24.244017 8476 ts_tablet_manager.cc:1428] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:24.244292 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.243451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.194:51961)
0924 23:16:24.243811 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.244250 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":80,"threads_started":1}
I20260924 23:16:24.244374 8481 tablet_bootstrap.cc:492] T 84d43c19453f4a40a4a81e3f5f4c6498 P 08ab7c555b634abda9f23607f5ee397c: No bootstrap required, opened a new log
I20260924 23:16:24.244407 8481 ts_tablet_manager.cc:1397] T 84d43c19453f4a40a4a81e3f5f4c6498 P 08ab7c555b634abda9f23607f5ee397c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:24.244513 8481 raft_consensus.cc:348] T 84d43c19453f4a40a4a81e3f5f4c6498 P 08ab7c555b634abda9f23607f5ee397c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:24.244585 8481 raft_consensus.cc:374] T 84d43c19453f4a40a4a81e3f5f4c6498 P 08ab7c555b634abda9f23607f5ee397c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:24.244587 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.244512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.194:36903)
0924 23:16:24.244546 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.244566 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 23:16:24.244606 8481 raft_consensus.cc:729] T 84d43c19453f4a40a4a81e3f5f4c6498 P 08ab7c555b634abda9f23607f5ee397c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ab7c555b634abda9f23607f5ee397c, State: Initialized, Role: FOLLOWER
I20260924 23:16:24.244645 8481 consensus_queue.cc:260] T 84d43c19453f4a40a4a81e3f5f4c6498 P 08ab7c555b634abda9f23607f5ee397c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:24.244714 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.244288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.195:37501)
0924 23:16:24.244471 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.244504 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":58,"threads_started":1}
I20260924 23:16:24.244764 8481 ts_tablet_manager.cc:1428] T 84d43c19453f4a40a4a81e3f5f4c6498 P 08ab7c555b634abda9f23607f5ee397c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:24.245097 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.244901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.195:36187)
0924 23:16:24.245029 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.245061 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260924 23:16:24.245143 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.244970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.193:38223)
0924 23:16:24.245081 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.245112 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":75,"threads_started":1}
I20260924 23:16:24.245591 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.245506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.193:35227)
0924 23:16:24.245536 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.245559 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 23:16:24.520920 8483 raft_consensus.cc:482] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:24.521018 8483 raft_consensus.cc:504] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:24.521308 8483 leader_election.cc:288] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08ab7c555b634abda9f23607f5ee397c (127.1.4.193:34339), e85bb432f60642fb92fa4c620b6ac7a0 (127.1.4.194:37679)
I20260924 23:16:24.524121 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.521554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:57073 (local address 127.1.4.193:34339)
0924 23:16:24.521644 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:24.521646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:24.521660 (+ 14us) server_negotiation.cc:407] Connection header received
0924 23:16:24.521725 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:24.521726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:24.521738 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:24.521785 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:24.521931 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.522413 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.523447 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.523806 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.523869 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:24.523882 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:24.524018 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 23:16:24.524052 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 23:16:24.524127 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.521514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:34339 (local address 127.1.4.195:57073)
0924 23:16:24.521575 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.521587 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:24.521653 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:24.521800 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:24.521802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:24.521805 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:24.521873 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:24.521874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.522497 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.522498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:24.523354 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:24.523357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.523879 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.523880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:24.523956 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:24.523961 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:16:24.524006 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 23:16:24.524046 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":28}
I20260924 23:16:24.524252 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.521626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:37679 (local address 127.1.4.195:52383)
0924 23:16:24.521750 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.521759 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:24.521816 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:24.521940 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:24.521940 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:24.521942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:24.522015 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:24.522016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.522652 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.522653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:24.523467 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:24.523469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.524034 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.524035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:24.524098 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:24.524102 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:24.524145 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 23:16:24.524182 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 23:16:24.524266 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.521718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:52383 (local address 127.1.4.194:37679)
0924 23:16:24.521770 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:24.521772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:24.521799 (+ 27us) server_negotiation.cc:407] Connection header received
0924 23:16:24.521873 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:24.521874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:24.521885 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:24.521930 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:24.522110 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.522577 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.523556 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.523951 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.524016 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:24.524029 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:24.524156 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 23:16:24.524192 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:16:24.524422 8065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d43c19453f4a40a4a81e3f5f4c6498" candidate_uuid: "3eec19b8e01544c7a91a6ab99e738b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ab7c555b634abda9f23607f5ee397c" is_pre_election: true
I20260924 23:16:24.524451 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d43c19453f4a40a4a81e3f5f4c6498" candidate_uuid: "3eec19b8e01544c7a91a6ab99e738b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" is_pre_election: true
I20260924 23:16:24.524549 8201 raft_consensus.cc:2454] T 84d43c19453f4a40a4a81e3f5f4c6498 P e85bb432f60642fb92fa4c620b6ac7a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eec19b8e01544c7a91a6ab99e738b73 in term 0.
I20260924 23:16:24.524560 8065 raft_consensus.cc:2454] T 84d43c19453f4a40a4a81e3f5f4c6498 P 08ab7c555b634abda9f23607f5ee397c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eec19b8e01544c7a91a6ab99e738b73 in term 0.
I20260924 23:16:24.524700 8272 leader_election.cc:302] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eec19b8e01544c7a91a6ab99e738b73, e85bb432f60642fb92fa4c620b6ac7a0; no voters:
I20260924 23:16:24.524823 8483 raft_consensus.cc:2781] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:24.524870 8483 raft_consensus.cc:482] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:24.524895 8483 raft_consensus.cc:3037] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:24.525552 8483 raft_consensus.cc:504] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:24.525663 8483 leader_election.cc:288] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [CANDIDATE]: Term 1 election: Requested vote from peers 08ab7c555b634abda9f23607f5ee397c (127.1.4.193:34339), e85bb432f60642fb92fa4c620b6ac7a0 (127.1.4.194:37679)
I20260924 23:16:24.525826 8065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d43c19453f4a40a4a81e3f5f4c6498" candidate_uuid: "3eec19b8e01544c7a91a6ab99e738b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ab7c555b634abda9f23607f5ee397c"
I20260924 23:16:24.525888 8065 raft_consensus.cc:3037] T 84d43c19453f4a40a4a81e3f5f4c6498 P 08ab7c555b634abda9f23607f5ee397c [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:24.525847 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d43c19453f4a40a4a81e3f5f4c6498" candidate_uuid: "3eec19b8e01544c7a91a6ab99e738b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85bb432f60642fb92fa4c620b6ac7a0"
I20260924 23:16:24.525918 8201 raft_consensus.cc:3037] T 84d43c19453f4a40a4a81e3f5f4c6498 P e85bb432f60642fb92fa4c620b6ac7a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:24.526489 8201 raft_consensus.cc:2454] T 84d43c19453f4a40a4a81e3f5f4c6498 P e85bb432f60642fb92fa4c620b6ac7a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eec19b8e01544c7a91a6ab99e738b73 in term 1.
I20260924 23:16:24.526489 8065 raft_consensus.cc:2454] T 84d43c19453f4a40a4a81e3f5f4c6498 P 08ab7c555b634abda9f23607f5ee397c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eec19b8e01544c7a91a6ab99e738b73 in term 1.
I20260924 23:16:24.526638 8272 leader_election.cc:302] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eec19b8e01544c7a91a6ab99e738b73, e85bb432f60642fb92fa4c620b6ac7a0; no voters:
I20260924 23:16:24.526714 8483 raft_consensus.cc:2781] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:24.526865 8483 raft_consensus.cc:686] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [term 1 LEADER]: Becoming Leader. State: Replica: 3eec19b8e01544c7a91a6ab99e738b73, State: Running, Role: LEADER
I20260924 23:16:24.526938 8385 heartbeater.cc:503] Master 127.1.4.253:45945 was elected leader, sending a full tablet report...
I20260924 23:16:24.526979 8483 consensus_queue.cc:237] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } } peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } }
I20260924 23:16:24.527158 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.527062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.195:57635)
0924 23:16:24.527104 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.527135 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 23:16:24.527717 7743 catalog_manager.cc:5495] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eec19b8e01544c7a91a6ab99e738b73 (127.1.4.195). New cstate: current_term: 1 leader_uuid: "3eec19b8e01544c7a91a6ab99e738b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eec19b8e01544c7a91a6ab99e738b73" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 } health_report { overall_health: UNKNOWN } } }
I20260924 23:16:24.592089 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.589764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.1.4.195:36637)
0924 23:16:24.589815 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:24.589817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:24.589825 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:24.589868 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:24.589868 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:24.589893 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:24.589962 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:24.590074 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.590553 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.591304 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:24.591559 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:24.591618 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:24.591770 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:24.591879 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:24.591950 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:24.591990 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 23:16:24.592011 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:16:24.593128 8201 raft_consensus.cc:1261] T 84d43c19453f4a40a4a81e3f5f4c6498 P e85bb432f60642fb92fa4c620b6ac7a0 [term 1 FOLLOWER]: Refusing update from remote peer 3eec19b8e01544c7a91a6ab99e738b73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:24.593225 8065 raft_consensus.cc:1261] T 84d43c19453f4a40a4a81e3f5f4c6498 P 08ab7c555b634abda9f23607f5ee397c [term 1 FOLLOWER]: Refusing update from remote peer 3eec19b8e01544c7a91a6ab99e738b73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:24.593385 8483 consensus_queue.cc:1035] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85bb432f60642fb92fa4c620b6ac7a0" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 37679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:24.593581 8483 consensus_queue.cc:1035] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08ab7c555b634abda9f23607f5ee397c" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:24.593977 8113 heartbeater.cc:503] Master 127.1.4.253:45945 was elected leader, sending a full tablet report...
I20260924 23:16:24.594497 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.594310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.194:57931)
0924 23:16:24.594418 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.594463 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 23:16:24.594599 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.594431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.193:36665)
0924 23:16:24.594539 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.594569 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 23:16:24.595631 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.595502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.194:46453)
0924 23:16:24.595574 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.595601 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 23:16:24.596804 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.596479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.193:46633)
0924 23:16:24.596747 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.596777 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251}
I20260924 23:16:24.597000 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.595610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42079 (local address 127.1.4.195:46215)
0924 23:16:24.596945 (+ 1335us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.596972 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1320}
I20260924 23:16:24.597707 7744 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:24.597779 7744 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:24.599016 7744 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b69cbd28a4394c14aaff8bb3901f5472: Sending DeleteTablet for 3 replicas of tablet 84d43c19453f4a40a4a81e3f5f4c6498
I20260924 23:16:24.599413 8317 tablet_service.cc:1514] Processing DeleteTablet for tablet 84d43c19453f4a40a4a81e3f5f4c6498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:24 UTC) from {username='oryx', principal='oryx/127.1.4.253@KRBTEST.COM'} at 127.0.0.1:41202
I20260924 23:16:24.599470 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 84d43c19453f4a40a4a81e3f5f4c6498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:24 UTC) from {username='oryx', principal='oryx/127.1.4.253@KRBTEST.COM'} at 127.0.0.1:49288
I20260924 23:16:24.599592 8501 tablet_replica.cc:330] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73: stopping tablet replica
I20260924 23:16:24.599640 8501 raft_consensus.cc:2229] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:16:24.599674 8045 tablet_service.cc:1514] Processing DeleteTablet for tablet 84d43c19453f4a40a4a81e3f5f4c6498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:24 UTC) from {username='oryx', principal='oryx/127.1.4.253@KRBTEST.COM'} at 127.0.0.1:47234
I20260924 23:16:24.599686 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 7982
I20260924 23:16:24.599717 8501 raft_consensus.cc:2258] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:24.601693 8501 ts_tablet_manager.cc:1905] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:24.603276 8501 ts_tablet_manager.cc:1918] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:24.603329 8501 log.cc:1196] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/84d43c19453f4a40a4a81e3f5f4c6498
I20260924 23:16:24.603562 8501 ts_tablet_manager.cc:1939] T 84d43c19453f4a40a4a81e3f5f4c6498 P 3eec19b8e01544c7a91a6ab99e738b73: Deleting consensus metadata
I20260924 23:16:24.603889 7730 catalog_manager.cc:4841] TS 3eec19b8e01544c7a91a6ab99e738b73 (127.1.4.195:36637): tablet 84d43c19453f4a40a4a81e3f5f4c6498 (table test-table [id=9be77a64628a4148809c6cf3e06e53c9]) successfully deleted
I20260924 23:16:24.605438 8502 tablet_replica.cc:330] T 84d43c19453f4a40a4a81e3f5f4c6498 P e85bb432f60642fb92fa4c620b6ac7a0: stopping tablet replica
I20260924 23:16:24.605439 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 8118
I20260924 23:16:24.605479 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.605375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:34339 (local address 127.0.0.1:47244)
0924 23:16:24.605423 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.605448 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:34339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 23:16:24.605511 8502 raft_consensus.cc:2229] T 84d43c19453f4a40a4a81e3f5f4c6498 P e85bb432f60642fb92fa4c620b6ac7a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:24.605558 8502 raft_consensus.cc:2258] T 84d43c19453f4a40a4a81e3f5f4c6498 P e85bb432f60642fb92fa4c620b6ac7a0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 23:16:24.605595 7730 catalog_manager.cc:4584] TS 08ab7c555b634abda9f23607f5ee397c (127.1.4.193:34339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84d43c19453f4a40a4a81e3f5f4c6498: Network error: Client connection negotiation failed: client connection to 127.1.4.193:34339: connect: Connection refused (error 111)
I20260924 23:16:24.610437 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 8254
I20260924 23:16:24.610492 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:24.610383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:37679 (local address 127.0.0.1:49302)
0924 23:16:24.610431 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:24.610462 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.194:37679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 23:16:24.614862 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 7611
I20260924 23:16:24.618883 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 7798
2026-09-24T23:16:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5162 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:16:24 dist-test-slave-s1hw krb5kdc[8505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:16:24 dist-test-slave-s1hw krb5kdc[8505](info): set up 2 sockets
Sep 24 23:16:24 dist-test-slave-s1hw krb5kdc[8505](info): commencing operation
krb5kdc: starting...
W20260924 23:16:25.571755 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:16:25 dist-test-slave-s1hw krb5kdc[8505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:16:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.254.keytab.
Entry for principal oryx/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.254.keytab.
W20260924 23:16:25.825804 1043 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.1.4.254: real 0.116s user 0.000s sys 0.002s
I20260924 23:16:25.826153 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38399
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.254.keytab
--principal=oryx/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:38399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:25.905630 8518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:25.905839 8518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:25.905862 8518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:25.906971 8518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:25.907003 8518 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 23:16:25.907015 8518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:25.907027 8518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:25.907037 8518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:25.907052 8518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:25.908524 8518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:38399
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.254.keytab
--principal=oryx/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:25.908716 8518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:25.908898 8518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:16:25.913681 8518 server_base.cc:1034] running on GCE node
W20260924 23:16:25.913579 8526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:25.913616 8523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:25.913582 8524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:25.914132 8518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:25.914402 8518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:25.915540 8518 hybrid_clock.cc:648] HybridClock initialized: now 1790291785915522 us; error 35 us; skew 500 ppm
Sep 24 23:16:25 dist-test-slave-s1hw krb5kdc[8505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291785, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:25.922145 8518 init.cc:377] Logged in from keytab as oryx/127.1.4.254@KRBTEST.COM (short username oryx)
I20260924 23:16:25.922585 8518 webserver.cc:460] Webserver started at http://127.1.4.254:38593/ using document root <none> and password file <none>
I20260924 23:16:25.922781 8518 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:25.922828 8518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:25.922993 8518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:25.924106 8518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "1caf36f04a2149de8e3586732b16db2d"
format_stamp: "Formatted at 2026-09-24 23:16:25 on dist-test-slave-s1hw"
server_key: "67db5b6230d8b2ced98f071dfe3b13e6"
server_key_iv: "62ec51fcefe0642040dccd43b23ba84b"
server_key_version: "encryptionkey@0"
I20260924 23:16:25.924440 8518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "1caf36f04a2149de8e3586732b16db2d"
format_stamp: "Formatted at 2026-09-24 23:16:25 on dist-test-slave-s1hw"
server_key: "67db5b6230d8b2ced98f071dfe3b13e6"
server_key_iv: "62ec51fcefe0642040dccd43b23ba84b"
server_key_version: "encryptionkey@0"
I20260924 23:16:25.925753 8518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:16:25.926574 8533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:25.926738 8518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:25.926826 8518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "1caf36f04a2149de8e3586732b16db2d"
format_stamp: "Formatted at 2026-09-24 23:16:25 on dist-test-slave-s1hw"
server_key: "67db5b6230d8b2ced98f071dfe3b13e6"
server_key_iv: "62ec51fcefe0642040dccd43b23ba84b"
server_key_version: "encryptionkey@0"
I20260924 23:16:25.926889 8518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:25.938172 8518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:25.940372 8518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:25.940521 8518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:25.944967 8518 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:38399
I20260924 23:16:25.945039 8595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:38399 every 8 connection(s)
I20260924 23:16:25.945343 8518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:25.946089 8596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:25.948408 8596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d: Bootstrap starting.
I20260924 23:16:25.948989 8596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:25.949250 8596 log.cc:824] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:25.949901 8596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d: No bootstrap required, opened a new log
I20260924 23:16:25.950770 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 8518
I20260924 23:16:25.950868 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:25.951068 1043 external_mini_cluster.cc:1428] Setting key 4df171481af298e4f3a52d37d41139cc
I20260924 23:16:25.952375 8596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caf36f04a2149de8e3586732b16db2d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38399 } }
I20260924 23:16:25.952519 8596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:25.952551 8596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1caf36f04a2149de8e3586732b16db2d, State: Initialized, Role: FOLLOWER
I20260924 23:16:25.952697 8596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caf36f04a2149de8e3586732b16db2d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38399 } }
I20260924 23:16:25.952766 8596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:25.952797 8596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:25.952843 8596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:25.953358 8596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caf36f04a2149de8e3586732b16db2d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38399 } }
I20260924 23:16:25.953460 8596 leader_election.cc:302] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1caf36f04a2149de8e3586732b16db2d; no voters:
I20260924 23:16:25.953614 8596 leader_election.cc:288] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:25.953653 8601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:25.953780 8601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [term 1 LEADER]: Becoming Leader. State: Replica: 1caf36f04a2149de8e3586732b16db2d, State: Running, Role: LEADER
I20260924 23:16:25.953873 8596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:25.953914 8601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caf36f04a2149de8e3586732b16db2d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38399 } }
I20260924 23:16:25.954276 8602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1caf36f04a2149de8e3586732b16db2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caf36f04a2149de8e3586732b16db2d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38399 } } }
I20260924 23:16:25.954350 8602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:25.954566 8603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1caf36f04a2149de8e3586732b16db2d. Latest consensus state: current_term: 1 leader_uuid: "1caf36f04a2149de8e3586732b16db2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1caf36f04a2149de8e3586732b16db2d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38399 } } }
I20260924 23:16:25.954864 8603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:25.955030 8607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:25.955678 8607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:25.957468 8607 catalog_manager.cc:1347] Generated new cluster ID: fd1e6bfb7f82405e9522379a5d06b0e0
I20260924 23:16:25.957521 8607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:16:25 dist-test-slave-s1hw krb5kdc[8505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM
I20260924 23:16:25.961450 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:25.951587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.1.4.254:38399)
0924 23:16:25.951775 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:25.951785 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:25.951816 (+ 31us) server_negotiation.cc:407] Connection header received
0924 23:16:25.952341 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:25.952347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:25.952394 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:25.952541 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:25.953192 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:25.954390 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:25.955459 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:25.955971 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:25.956065 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:25.958362 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:25.958367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:25.958367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:25.958380 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:25.960025 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:25.960330 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:25.960333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:25.960334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:25.960425 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:25.960592 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:25.960593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:25.960594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:25.960829 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:25.960987 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:25.961085 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 23:16:25.961303 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260924 23:16:25.989157 8607 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:25.989681 8607 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:25.995623 8607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d: Generated new TSK 0
I20260924 23:16:25.995807 8607 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.193.keytab.
Entry for principal oryx/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.193.keytab.
I20260924 23:16:26.030638 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38399
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.193.keytab
--principal=oryx/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:26.109385 8622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:26.109606 8622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:26.109629 8622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:26.110767 8622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:26.110802 8622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:26.110821 8622 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:16:26.110836 8622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:16:26.112367 8622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.193.keytab
--principal=oryx/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.4.254:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:26.112561 8622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:26.112723 8622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:26.117451 8630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:26.117461 8628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:26.117487 8622 server_base.cc:1034] running on GCE node
W20260924 23:16:26.117461 8627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:26.117875 8622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:26.118088 8622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:26.119222 8622 hybrid_clock.cc:648] HybridClock initialized: now 1790291786119205 us; error 33 us; skew 500 ppm
Sep 24 23:16:26 dist-test-slave-s1hw krb5kdc[8505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291786, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:26.121591 8622 init.cc:377] Logged in from keytab as oryx/127.1.4.193@KRBTEST.COM (short username oryx)
I20260924 23:16:26.122010 8622 webserver.cc:460] Webserver started at http://127.1.4.193:34237/ using document root <none> and password file <none>
I20260924 23:16:26.122202 8622 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:26.122246 8622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:26.122350 8622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:26.123447 8622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "9fbdf25c612649c9a6506b26fc3db913"
format_stamp: "Formatted at 2026-09-24 23:16:26 on dist-test-slave-s1hw"
server_key: "4b70eff435126170367d49cdd2ec0367"
server_key_iv: "ac4838a621e68a18da6eeb2f8b7c7481"
server_key_version: "encryptionkey@0"
I20260924 23:16:26.123768 8622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "9fbdf25c612649c9a6506b26fc3db913"
format_stamp: "Formatted at 2026-09-24 23:16:26 on dist-test-slave-s1hw"
server_key: "4b70eff435126170367d49cdd2ec0367"
server_key_iv: "ac4838a621e68a18da6eeb2f8b7c7481"
server_key_version: "encryptionkey@0"
I20260924 23:16:26.125162 8622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:16:26.125927 8637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:26.126107 8622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:16:26.126187 8622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "9fbdf25c612649c9a6506b26fc3db913"
format_stamp: "Formatted at 2026-09-24 23:16:26 on dist-test-slave-s1hw"
server_key: "4b70eff435126170367d49cdd2ec0367"
server_key_iv: "ac4838a621e68a18da6eeb2f8b7c7481"
server_key_version: "encryptionkey@0"
I20260924 23:16:26.126247 8622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:26.133942 8622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:26.135927 8622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:26.136078 8622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:26.136569 8622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:26.136611 8622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:26.136642 8622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:26.136668 8622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 23:16:26 dist-test-slave-s1hw krb5kdc[8505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291786, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM
I20260924 23:16:26.143174 8622 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:42525
I20260924 23:16:26.143257 8752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:42525 every 8 connection(s)
I20260924 23:16:26.143586 8622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:16:26.145035 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 8622
I20260924 23:16:26.145117 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:16:26.145295 1043 external_mini_cluster.cc:1428] Setting key 615ac5de1f384b5a1c5763e7f8c6294d
I20260924 23:16:26.146224 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.137042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:50445 (local address 127.1.4.254:38399)
0924 23:16:26.137107 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:26.137109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:26.137318 (+ 209us) server_negotiation.cc:407] Connection header received
0924 23:16:26.137830 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:26.137832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:26.137849 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:26.137931 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:26.138722 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.139499 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.140515 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.140830 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.140904 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.143352 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:26.143354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:26.143354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:26.143390 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:26.144764 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:26.145196 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:26.145197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:26.145198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:26.145257 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:26.145483 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:26.145485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:26.145485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:26.145661 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:26.145726 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:26.146029 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 23:16:26.146136 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 23:16:26.146368 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.137021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38399 (local address 127.1.4.193:50445)
0924 23:16:26.137186 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:26.137231 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:26.137732 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:26.138002 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:26.138007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:26.138340 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:26.138626 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.138632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.139586 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.139587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.140446 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.140449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.140987 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.140988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.141145 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.141709 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:26.141718 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:26.143259 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:26.144841 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:26.144846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:26.144848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:26.145130 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:26.145311 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:26.145312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:26.145312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:26.145426 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:26.145737 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:26.145743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:26.145892 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 23:16:26.145997 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 23:16:26.146159 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260924 23:16:26.147557 8753 heartbeater.cc:347] Connected to a master server at 127.1.4.254:38399
I20260924 23:16:26.147650 8753 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:26.147874 8753 heartbeater.cc:511] Master 127.1.4.254:38399 requested a full tablet report, sending...
I20260924 23:16:26.148396 8549 ts_manager.cc:194] Registered new tserver with Master: 9fbdf25c612649c9a6506b26fc3db913 (127.1.4.193:42525)
I20260924 23:16:26.149099 8549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:50445
WARNING: no policy specified for oryx/127.1.4.194@KRBTEST.COM; defaulting to no policy
I20260924 23:16:26.155093 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.148660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38399 (local address 127.1.4.193:46281)
0924 23:16:26.149368 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:26.149378 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:26.149427 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:26.149532 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:26.149533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:26.149732 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:26.149794 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.149796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.150739 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.150740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.151254 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.151256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.151728 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.151729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.151781 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.152227 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:26.152231 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:26.152521 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:26.154168 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:26.154169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:26.154170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:26.154334 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:26.154512 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:26.154513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:26.154513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:26.154561 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:26.154800 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:26.154801 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:26.154841 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 23:16:26.154891 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 23:16:26.154976 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":72,"threads_started":1}
I20260924 23:16:26.155138 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.148725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:46281 (local address 127.1.4.254:38399)
0924 23:16:26.148761 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:26.148763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:26.149431 (+ 668us) server_negotiation.cc:407] Connection header received
0924 23:16:26.149479 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:26.149480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:26.149490 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:26.149530 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:26.149877 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.150643 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.151333 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.151661 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.151732 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.152611 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:26.152612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:26.152613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:26.152620 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:26.154067 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:26.154416 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:26.154417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:26.154417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:26.154465 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:26.154594 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:26.154594 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:26.154595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:26.154716 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:26.154771 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:26.154938 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 23:16:26.155042 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.4.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.194.keytab.
Entry for principal oryx/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.194.keytab.
I20260924 23:16:26.170832 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38399
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.194.keytab
--principal=oryx/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:26.252826 8762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:26.253041 8762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:26.253064 8762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:26.254346 8762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:26.254398 8762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:26.254418 8762 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:16:26.254434 8762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:16:26.256078 8762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.194.keytab
--principal=oryx/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.4.254:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:26.256301 8762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:26.256516 8762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:26.261297 8767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:26.261297 8770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:26.261370 8762 server_base.cc:1034] running on GCE node
W20260924 23:16:26.261307 8768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:26.261704 8762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:26.261927 8762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:26.263063 8762 hybrid_clock.cc:648] HybridClock initialized: now 1790291786263054 us; error 26 us; skew 500 ppm
Sep 24 23:16:26 dist-test-slave-s1hw krb5kdc[8505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291786, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:26.265381 8762 init.cc:377] Logged in from keytab as oryx/127.1.4.194@KRBTEST.COM (short username oryx)
I20260924 23:16:26.265798 8762 webserver.cc:460] Webserver started at http://127.1.4.194:38869/ using document root <none> and password file <none>
I20260924 23:16:26.265996 8762 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:26.266036 8762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:26.266129 8762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:26.267107 8762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "a4aee7e753324811a3ad389c9e942776"
format_stamp: "Formatted at 2026-09-24 23:16:26 on dist-test-slave-s1hw"
server_key: "cc96ef2822e90547e3d374175e2da386"
server_key_iv: "97cd4faf6981ef4bfed8ea3b8bce6a82"
server_key_version: "encryptionkey@0"
I20260924 23:16:26.267433 8762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "a4aee7e753324811a3ad389c9e942776"
format_stamp: "Formatted at 2026-09-24 23:16:26 on dist-test-slave-s1hw"
server_key: "cc96ef2822e90547e3d374175e2da386"
server_key_iv: "97cd4faf6981ef4bfed8ea3b8bce6a82"
server_key_version: "encryptionkey@0"
I20260924 23:16:26.268620 8762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:16:26.269354 8777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:26.269521 8762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:16:26.269595 8762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "a4aee7e753324811a3ad389c9e942776"
format_stamp: "Formatted at 2026-09-24 23:16:26 on dist-test-slave-s1hw"
server_key: "cc96ef2822e90547e3d374175e2da386"
server_key_iv: "97cd4faf6981ef4bfed8ea3b8bce6a82"
server_key_version: "encryptionkey@0"
I20260924 23:16:26.269654 8762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:26.276450 8762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:26.278280 8762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:26.278411 8762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:26.278904 8762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:26.278947 8762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:26.278995 8762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:26.279018 8762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:26.285247 8762 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:42881
I20260924 23:16:26.285286 8892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:42881 every 8 connection(s)
I20260924 23:16:26.285593 8762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:16:26 dist-test-slave-s1hw krb5kdc[8505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291786, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM
I20260924 23:16:26.288771 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.279427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:39779 (local address 127.1.4.254:38399)
0924 23:16:26.279475 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:26.279476 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:26.279687 (+ 211us) server_negotiation.cc:407] Connection header received
0924 23:16:26.280277 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:26.280279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:26.280292 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:26.280345 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:26.281135 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.281886 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.283453 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.283650 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.283706 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.286242 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:26.286244 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:26.286244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:26.286252 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:26.287539 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:26.287901 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:26.287902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:26.287903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:26.287956 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:26.288171 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:26.288172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:26.288172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:26.288286 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:26.288340 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:26.288600 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 23:16:26.288679 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:16:26.288947 8786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.279425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38399 (local address 127.1.4.194:39779)
0924 23:16:26.279578 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:26.279611 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:26.280183 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:26.280419 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:26.280423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:26.280756 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:26.281054 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.281062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.281990 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.281991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.283310 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.283314 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.283738 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.283739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.283893 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.284471 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:26.284490 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:26.286147 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:26.287609 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:26.287617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:26.287619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:26.287834 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:26.288011 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:26.288011 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:26.288012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:26.288118 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:26.288337 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:26.288344 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:26.288485 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 23:16:26.288575 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 23:16:26.288733 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260924 23:16:26.289443 8893 heartbeater.cc:347] Connected to a master server at 127.1.4.254:38399
I20260924 23:16:26.289530 8893 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:26.289695 8893 heartbeater.cc:511] Master 127.1.4.254:38399 requested a full tablet report, sending...
I20260924 23:16:26.290092 8549 ts_manager.cc:194] Registered new tserver with Master: a4aee7e753324811a3ad389c9e942776 (127.1.4.194:42881)
I20260924 23:16:26.290557 8549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:39779
I20260924 23:16:26.295608 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 8762
I20260924 23:16:26.295720 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:16:26.295759 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.290454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38399 (local address 127.1.4.194:34067)
0924 23:16:26.290588 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:26.290599 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:26.290661 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:26.290791 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:26.290792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:26.290984 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:26.291056 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.291059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.291764 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.291765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.292264 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.292265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.292587 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.292588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.292662 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.293071 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:26.293075 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:26.293339 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:26.294739 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:26.294740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:26.294741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:26.294896 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:26.295089 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:26.295090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:26.295090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:26.295152 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:26.295460 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:26.295460 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:26.295504 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 23:16:26.295567 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 23:16:26.295652 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":4,"thread_start_us":64,"threads_started":1}
I20260924 23:16:26.295764 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.290531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:34067 (local address 127.1.4.254:38399)
0924 23:16:26.290572 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:26.290573 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:26.290689 (+ 116us) server_negotiation.cc:407] Connection header received
0924 23:16:26.290714 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:26.290714 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:26.290724 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:26.290782 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:26.291128 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.291664 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.292328 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.292507 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.292554 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.293443 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:26.293445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:26.293445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:26.293451 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:26.294672 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:26.294954 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:26.294954 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:26.294955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:26.295018 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:26.295224 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:26.295224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:26.295225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:26.295360 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:26.295464 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:26.295604 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 23:16:26.295678 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:16:26.295934 1043 external_mini_cluster.cc:1428] Setting key e6bcc50208c32f6dc9f95e3d740789ac
WARNING: no policy specified for oryx/127.1.4.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.195.keytab.
Entry for principal oryx/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.195.keytab.
I20260924 23:16:26.358860 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38399
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.195.keytab
--principal=oryx/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:26.438879 8902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:26.439110 8902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:26.439136 8902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:26.440322 8902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:26.440371 8902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:26.440392 8902 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:16:26.440407 8902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:16:26.441955 8902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.195.keytab
--principal=oryx/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.4.254:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:26.442178 8902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:26.442373 8902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:26.447197 8908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:26.447204 8907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:26.447221 8910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:26.447484 8902 server_base.cc:1034] running on GCE node
I20260924 23:16:26.447674 8902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:26.447953 8902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:26.449136 8902 hybrid_clock.cc:648] HybridClock initialized: now 1790291786449106 us; error 28 us; skew 500 ppm
Sep 24 23:16:26 dist-test-slave-s1hw krb5kdc[8505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291786, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:26.451766 8902 init.cc:377] Logged in from keytab as oryx/127.1.4.195@KRBTEST.COM (short username oryx)
I20260924 23:16:26.452243 8902 webserver.cc:460] Webserver started at http://127.1.4.195:35353/ using document root <none> and password file <none>
I20260924 23:16:26.452497 8902 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:26.452549 8902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:26.452675 8902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:26.453780 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "24665fd26a98463d864d9dcdb6321959"
format_stamp: "Formatted at 2026-09-24 23:16:26 on dist-test-slave-s1hw"
server_key: "6691a29efee9325b1fde88f4edd6cfff"
server_key_iv: "8725bec143a0b7d11fdb2ff9b743d778"
server_key_version: "encryptionkey@0"
I20260924 23:16:26.454102 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "24665fd26a98463d864d9dcdb6321959"
format_stamp: "Formatted at 2026-09-24 23:16:26 on dist-test-slave-s1hw"
server_key: "6691a29efee9325b1fde88f4edd6cfff"
server_key_iv: "8725bec143a0b7d11fdb2ff9b743d778"
server_key_version: "encryptionkey@0"
I20260924 23:16:26.455545 8902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:26.456247 8917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:26.456418 8902 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:26.456501 8902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "24665fd26a98463d864d9dcdb6321959"
format_stamp: "Formatted at 2026-09-24 23:16:26 on dist-test-slave-s1hw"
server_key: "6691a29efee9325b1fde88f4edd6cfff"
server_key_iv: "8725bec143a0b7d11fdb2ff9b743d778"
server_key_version: "encryptionkey@0"
I20260924 23:16:26.456560 8902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:26.471601 8902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:26.473517 8902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:26.473659 8902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:26.474159 8902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:26.474202 8902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:26.474231 8902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:26.474254 8902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 23:16:26 dist-test-slave-s1hw krb5kdc[8505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291786, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM
I20260924 23:16:26.480860 8902 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:37963
I20260924 23:16:26.480921 9032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:37963 every 8 connection(s)
I20260924 23:16:26.481262 8902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:16:26.483489 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 8902
I20260924 23:16:26.483564 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.474558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:50323 (local address 127.1.4.254:38399)
0924 23:16:26.474608 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:26.474610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:26.474798 (+ 188us) server_negotiation.cc:407] Connection header received
0924 23:16:26.475311 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:26.475313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:26.475326 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:26.475396 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:26.476100 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.476596 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.477645 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.477863 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.477920 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.480750 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:26.480753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:26.480753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:26.480762 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:26.482295 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:26.482728 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:26.482729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:26.482730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:26.482783 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:26.482956 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:26.482957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:26.482957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:26.483067 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:26.483115 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:26.483360 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 23:16:26.483460 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 23:16:26.483593 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:16:26.483778 1043 external_mini_cluster.cc:1428] Setting key 4cbb88b4d4c3187135f4a2dec7fce5d5
I20260924 23:16:26.483770 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.474555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38399 (local address 127.1.4.195:50323)
0924 23:16:26.474686 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:26.474722 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:26.475215 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:26.475474 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:26.475477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:26.475797 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:26.476004 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.476009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.476699 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.476700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.477559 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.477561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.477933 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.477934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.478072 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.478550 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:26.478561 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:26.480625 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:26.482388 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:26.482395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:26.482396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:26.482654 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:26.482826 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:26.482826 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:26.482827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:26.482915 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:26.483118 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:26.483121 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:26.483241 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 23:16:26.483325 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 23:16:26.483505 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260924 23:16:26.484256 9033 heartbeater.cc:347] Connected to a master server at 127.1.4.254:38399
I20260924 23:16:26.484366 9033 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:26.484611 9033 heartbeater.cc:511] Master 127.1.4.254:38399 requested a full tablet report, sending...
I20260924 23:16:26.484977 8548 ts_manager.cc:194] Registered new tserver with Master: 24665fd26a98463d864d9dcdb6321959 (127.1.4.195:37963)
I20260924 23:16:26.485446 8548 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:50323
I20260924 23:16:26.490615 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.485301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:56713 (local address 127.1.4.254:38399)
0924 23:16:26.485335 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:26.485337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:26.485645 (+ 308us) server_negotiation.cc:407] Connection header received
0924 23:16:26.485710 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:26.485710 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:26.485718 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:26.485761 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:26.486082 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.486571 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.487241 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.487507 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.487556 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.488349 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:26.488351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:26.488351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:26.488357 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:26.489619 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:26.489913 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:26.489914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:26.489914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:26.489968 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:26.490128 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:26.490129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:26.490129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:26.490233 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:26.490284 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:26.490416 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 23:16:26.490502 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 23:16:26.490633 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.485245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38399 (local address 127.1.4.195:56713)
0924 23:16:26.485589 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:26.485600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:26.485658 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:26.485778 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:26.485779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:26.485951 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:26.486016 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.486018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.486653 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.486654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.487173 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.487174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.487571 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.487572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.487637 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.488007 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:26.488011 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:26.488273 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:26.489686 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:26.489687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:26.489688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:26.489846 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:26.490020 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:26.490021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:26.490022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:26.490077 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:26.490283 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:26.490284 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:26.490327 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 23:16:26.490387 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 23:16:26.490517 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":62,"threads_started":1}
I20260924 23:16:26.495646 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 23:16:26 dist-test-slave-s1hw krb5kdc[8505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 23:16:26 dist-test-slave-s1hw krb5kdc[8505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:26 dist-test-slave-s1hw krb5kdc[8505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 23:16:26.515287 8600 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:26.509847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.1.4.254:38399)
0924 23:16:26.509899 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:26.509901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:26.509979 (+ 78us) server_negotiation.cc:407] Connection header received
0924 23:16:26.510036 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:26.510037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:26.510045 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:26.510090 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:26.510664 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.511204 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.512007 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.512184 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.512231 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.515217 (+ 2986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56758: BlockingRecv error: recv got EOF from 127.0.0.1:56758 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 23:16:26 dist-test-slave-s1hw krb5kdc[8505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM
I20260924 23:16:26.522285 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.515992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56762 (local address 127.1.4.254:38399)
0924 23:16:26.516031 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:26.516033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:26.516069 (+ 36us) server_negotiation.cc:407] Connection header received
0924 23:16:26.516126 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:26.516127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:26.516135 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:26.516175 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:26.516578 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.517046 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.517728 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.517897 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.517938 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.520032 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:26.520033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:26.520033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:26.520039 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:26.521271 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:26.521588 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:26.521589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:26.521589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:26.521637 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:26.521848 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:26.521849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:26.521849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:26.521945 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:26.522013 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:26.522156 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 23:16:26.522214 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 23:16:26.523964 8548 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:16:26.525533 8548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:26.532228 9055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.527817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:42881 (local address 127.0.0.1:32796)
0924 23:16:26.528556 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:26.528567 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:26.528616 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:26.528991 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:26.528992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:26.528994 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:26.529068 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.529070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.530192 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.530192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.531277 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.531278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.532030 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.532031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.532086 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.532089 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:26.532129 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 23:16:26.532157 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":3,"thread_start_us":171,"threads_started":1}
I20260924 23:16:26.532331 9059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.528054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57660 (local address 127.1.4.195:37963)
0924 23:16:26.528237 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:26.528242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:26.528253 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:16:26.528314 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:26.528316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:26.528333 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:26.528466 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:26.528655 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.529793 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.531099 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.531828 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.531907 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.531989 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:26.532199 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 23:16:26.532236 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":86,"threads_started":1}
I20260924 23:16:26.532330 9056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.527816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:42525 (local address 127.0.0.1:47426)
0924 23:16:26.527960 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:26.528013 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:26.528076 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:26.528467 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:26.528470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:26.528473 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:26.528574 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.528576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.529910 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.529911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.531193 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.531196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.532068 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.532069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.532141 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.532145 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:26.532186 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 23:16:26.532228 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":41,"thread_start_us":95,"threads_started":1}
I20260924 23:16:26.532554 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.528133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32796 (local address 127.1.4.194:42881)
0924 23:16:26.528289 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:26.528294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:26.528669 (+ 375us) server_negotiation.cc:407] Connection header received
0924 23:16:26.528719 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:26.528721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:26.528736 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:26.528804 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:26.529163 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.530128 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.531330 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.531852 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.531915 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.531999 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:26.532415 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 23:16:26.532456 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260924 23:16:26.532660 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.527817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:37963 (local address 127.0.0.1:57660)
0924 23:16:26.528158 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:26.528166 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:26.528202 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:26.528473 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:26.528474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:26.528475 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:26.528576 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.528577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.529883 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.529884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.531019 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.531022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.531913 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.531914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.531989 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.532004 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 23:16:26.532070 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 23:16:26.532126 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":37,"thread_start_us":53,"threads_started":1}
I20260924 23:16:26.533258 8967 tablet_service.cc:1467] Processing CreateTablet for tablet 133ce4d0c83d48f39b8f733ed718c401 (DEFAULT_TABLE table=test-table [id=ef0c7815d88a4c07bb0fc874bfe9176c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:26.533257 8810 tablet_service.cc:1467] Processing CreateTablet for tablet 133ce4d0c83d48f39b8f733ed718c401 (DEFAULT_TABLE table=test-table [id=ef0c7815d88a4c07bb0fc874bfe9176c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:26.533565 8810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133ce4d0c83d48f39b8f733ed718c401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:26.533569 8967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133ce4d0c83d48f39b8f733ed718c401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:26.536302 9061 tablet_bootstrap.cc:492] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959: Bootstrap starting.
I20260924 23:16:26.536604 9062 tablet_bootstrap.cc:492] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776: Bootstrap starting.
I20260924 23:16:26.536710 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.528002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.1.4.193:42525)
0924 23:16:26.528312 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:26.528315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:26.528323 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:26.528355 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:26.528358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:26.528371 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:26.528448 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:26.528655 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.529854 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.531276 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.531998 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.532054 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.532121 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:26.532209 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 23:16:26.532243 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":76,"threads_started":1}
I20260924 23:16:26.536957 9061 tablet_bootstrap.cc:654] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:26.537191 9062 tablet_bootstrap.cc:654] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:26.537267 9061 log.cc:824] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:26.537462 9062 log.cc:824] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:26.537436 8687 tablet_service.cc:1467] Processing CreateTablet for tablet 133ce4d0c83d48f39b8f733ed718c401 (DEFAULT_TABLE table=test-table [id=ef0c7815d88a4c07bb0fc874bfe9176c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:26.537752 8687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133ce4d0c83d48f39b8f733ed718c401. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:26.537953 9061 tablet_bootstrap.cc:492] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959: No bootstrap required, opened a new log
I20260924 23:16:26.538020 9061 ts_tablet_manager.cc:1397] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:26.538190 9062 tablet_bootstrap.cc:492] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776: No bootstrap required, opened a new log
I20260924 23:16:26.538239 9062 ts_tablet_manager.cc:1397] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:26.539902 9061 raft_consensus.cc:348] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } }
I20260924 23:16:26.540074 9061 raft_consensus.cc:374] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:26.540108 9061 raft_consensus.cc:729] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24665fd26a98463d864d9dcdb6321959, State: Initialized, Role: FOLLOWER
I20260924 23:16:26.540068 9062 raft_consensus.cc:348] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } }
I20260924 23:16:26.540192 9062 raft_consensus.cc:374] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:26.540220 9062 raft_consensus.cc:729] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4aee7e753324811a3ad389c9e942776, State: Initialized, Role: FOLLOWER
I20260924 23:16:26.540225 9061 consensus_queue.cc:260] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } }
I20260924 23:16:26.540364 9062 consensus_queue.cc:260] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } }
I20260924 23:16:26.540481 9061 ts_tablet_manager.cc:1428] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:16:26.540524 9033 heartbeater.cc:503] Master 127.1.4.254:38399 was elected leader, sending a full tablet report...
I20260924 23:16:26.540612 9062 ts_tablet_manager.cc:1428] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:16:26.540673 8893 heartbeater.cc:503] Master 127.1.4.254:38399 was elected leader, sending a full tablet report...
I20260924 23:16:26.541632 9067 tablet_bootstrap.cc:492] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913: Bootstrap starting.
I20260924 23:16:26.542269 9067 tablet_bootstrap.cc:654] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:26.542511 9067 log.cc:824] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:26.543125 9067 tablet_bootstrap.cc:492] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913: No bootstrap required, opened a new log
I20260924 23:16:26.543179 9067 ts_tablet_manager.cc:1397] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:26.544691 9067 raft_consensus.cc:348] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } }
I20260924 23:16:26.544827 9067 raft_consensus.cc:374] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:26.544847 9067 raft_consensus.cc:729] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbdf25c612649c9a6506b26fc3db913, State: Initialized, Role: FOLLOWER
I20260924 23:16:26.544935 9067 consensus_queue.cc:260] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } }
I20260924 23:16:26.545161 9067 ts_tablet_manager.cc:1428] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:16:26.545207 8753 heartbeater.cc:503] Master 127.1.4.254:38399 was elected leader, sending a full tablet report...
W20260924 23:16:26.644295 8754 tablet.cc:2369] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:16:26.731846 9034 tablet.cc:2369] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:16:26.786334 8894 tablet.cc:2369] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:16:26.905092 9069 raft_consensus.cc:482] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:26.905216 9069 raft_consensus.cc:504] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } }
I20260924 23:16:26.905697 9069 leader_election.cc:288] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4aee7e753324811a3ad389c9e942776 (127.1.4.194:42881), 24665fd26a98463d864d9dcdb6321959 (127.1.4.195:37963)
I20260924 23:16:26.908442 9059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.906060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:59595 (local address 127.1.4.195:37963)
0924 23:16:26.906103 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:26.906105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:26.906113 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:26.906149 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:26.906149 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:26.906159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:26.906203 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:26.906346 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.906800 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.907767 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.908114 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.908173 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.908183 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:26.908324 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 23:16:26.908362 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:16:26.908473 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.905865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:37963 (local address 127.1.4.193:59595)
0924 23:16:26.905988 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:26.906003 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:26.906077 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:26.906212 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:26.906213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:26.906215 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:26.906281 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.906283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.906869 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.906870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.907688 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.907690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.908179 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.908179 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.908240 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.908244 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:26.908309 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 23:16:26.908362 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":1,"thread_start_us":65,"threads_started":1}
I20260924 23:16:26.908488 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.906023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:43843 (local address 127.1.4.194:42881)
0924 23:16:26.906078 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:26.906080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:26.906095 (+ 15us) server_negotiation.cc:407] Connection header received
0924 23:16:26.906175 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:26.906176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:26.906187 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:26.906231 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:26.906388 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.906834 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.907771 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.908134 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.908234 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.908244 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:26.908396 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 23:16:26.908425 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 23:16:26.908703 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.905941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:42881 (local address 127.1.4.193:43843)
0924 23:16:26.906040 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:26.906050 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:26.906106 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:26.906242 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:26.906242 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:26.906244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:26.906299 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.906301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.906907 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.906907 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.907694 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:26.907696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.908248 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.908249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:26.908327 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.908330 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:26.908375 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 23:16:26.908420 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":2,"thread_start_us":41,"threads_started":1}
I20260924 23:16:26.908859 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133ce4d0c83d48f39b8f733ed718c401" candidate_uuid: "9fbdf25c612649c9a6506b26fc3db913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24665fd26a98463d864d9dcdb6321959" is_pre_election: true
I20260924 23:16:26.909009 8987 raft_consensus.cc:2454] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbdf25c612649c9a6506b26fc3db913 in term 0.
I20260924 23:16:26.909219 8638 leader_election.cc:302] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24665fd26a98463d864d9dcdb6321959, 9fbdf25c612649c9a6506b26fc3db913; no voters:
I20260924 23:16:26.909245 8847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133ce4d0c83d48f39b8f733ed718c401" candidate_uuid: "9fbdf25c612649c9a6506b26fc3db913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4aee7e753324811a3ad389c9e942776" is_pre_election: true
I20260924 23:16:26.909368 8847 raft_consensus.cc:2454] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbdf25c612649c9a6506b26fc3db913 in term 0.
I20260924 23:16:26.909343 9069 raft_consensus.cc:2781] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:26.909407 9069 raft_consensus.cc:482] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:26.909438 9069 raft_consensus.cc:3037] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:26.910118 9069 raft_consensus.cc:504] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } }
I20260924 23:16:26.910228 9069 leader_election.cc:288] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [CANDIDATE]: Term 1 election: Requested vote from peers a4aee7e753324811a3ad389c9e942776 (127.1.4.194:42881), 24665fd26a98463d864d9dcdb6321959 (127.1.4.195:37963)
I20260924 23:16:26.910372 8847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133ce4d0c83d48f39b8f733ed718c401" candidate_uuid: "9fbdf25c612649c9a6506b26fc3db913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4aee7e753324811a3ad389c9e942776"
I20260924 23:16:26.910405 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133ce4d0c83d48f39b8f733ed718c401" candidate_uuid: "9fbdf25c612649c9a6506b26fc3db913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24665fd26a98463d864d9dcdb6321959"
I20260924 23:16:26.910442 8847 raft_consensus.cc:3037] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:26.910487 8987 raft_consensus.cc:3037] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:26.911142 8847 raft_consensus.cc:2454] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbdf25c612649c9a6506b26fc3db913 in term 1.
I20260924 23:16:26.911281 8987 raft_consensus.cc:2454] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbdf25c612649c9a6506b26fc3db913 in term 1.
I20260924 23:16:26.911316 8640 leader_election.cc:302] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fbdf25c612649c9a6506b26fc3db913, a4aee7e753324811a3ad389c9e942776; no voters:
I20260924 23:16:26.911439 9069 raft_consensus.cc:2781] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:26.911535 9069 raft_consensus.cc:686] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 LEADER]: Becoming Leader. State: Replica: 9fbdf25c612649c9a6506b26fc3db913, State: Running, Role: LEADER
I20260924 23:16:26.911619 9069 consensus_queue.cc:237] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } }
I20260924 23:16:26.912344 8550 catalog_manager.cc:5495] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fbdf25c612649c9a6506b26fc3db913 (127.1.4.193). New cstate: current_term: 1 leader_uuid: "9fbdf25c612649c9a6506b26fc3db913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } health_report { overall_health: HEALTHY } } }
I20260924 23:16:26.972043 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:26.969769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47440 (local address 127.1.4.193:42525)
0924 23:16:26.969829 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:26.969831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:26.969839 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:26.969873 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:26.969874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:26.969892 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:26.969955 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:26.970064 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.970573 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.971344 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:26.971582 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:26.971640 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:26.971777 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:26.971878 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:26.971932 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:26.971960 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 23:16:26.971977 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 23:16:26.973861 8987 raft_consensus.cc:1261] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959 [term 1 FOLLOWER]: Refusing update from remote peer 9fbdf25c612649c9a6506b26fc3db913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:26.973861 8847 raft_consensus.cc:1261] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Refusing update from remote peer 9fbdf25c612649c9a6506b26fc3db913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:26.974174 9072 consensus_queue.cc:1035] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:26.974305 9069 consensus_queue.cc:1035] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:26.977350 9077 mvcc.cc:205] Tried to move back new op lower bound from 7333035159442296832 to 7333035159190388736. Current Snapshot: MvccSnapshot[applied={T|T < 7333035159442296832}]
I20260924 23:16:26.978435 8550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:26.978528 8550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:26.979269 8550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d: Sending DeleteTablet for 3 replicas of tablet 133ce4d0c83d48f39b8f733ed718c401
I20260924 23:16:26.979607 8810 tablet_service.cc:1514] Processing DeleteTablet for tablet 133ce4d0c83d48f39b8f733ed718c401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:26 UTC) from {username='oryx', principal='oryx/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:32796
I20260924 23:16:26.979637 8687 tablet_service.cc:1514] Processing DeleteTablet for tablet 133ce4d0c83d48f39b8f733ed718c401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:26 UTC) from {username='oryx', principal='oryx/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:47426
I20260924 23:16:26.979976 9084 tablet_replica.cc:330] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776: stopping tablet replica
I20260924 23:16:26.979938 8550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:16:26.980082 8550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:26.980082 9084 raft_consensus.cc:2229] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:26.980176 9084 raft_consensus.cc:2258] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:26.980269 8967 tablet_service.cc:1514] Processing DeleteTablet for tablet 133ce4d0c83d48f39b8f733ed718c401 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:26 UTC) from {username='oryx', principal='oryx/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:57660
I20260924 23:16:26.980511 9085 tablet_replica.cc:330] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913: stopping tablet replica
I20260924 23:16:26.980630 9085 raft_consensus.cc:2229] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:16:26.980693 9086 tablet_replica.cc:330] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959: stopping tablet replica
I20260924 23:16:26.980758 9085 raft_consensus.cc:2258] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:26.980829 9086 raft_consensus.cc:2229] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:26.981101 9086 raft_consensus.cc:2258] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:26.981261 9085 ts_tablet_manager.cc:1905] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:26.981812 8687 tablet_service.cc:1467] Processing CreateTablet for tablet 0f83153bc55544cba051f923bf8a29a8 (DEFAULT_TABLE table=test-table [id=9acc228ece964de590309b79b9ac8ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:26.981848 8967 tablet_service.cc:1467] Processing CreateTablet for tablet 0f83153bc55544cba051f923bf8a29a8 (DEFAULT_TABLE table=test-table [id=9acc228ece964de590309b79b9ac8ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:26.981943 8687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f83153bc55544cba051f923bf8a29a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:26.981977 8967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f83153bc55544cba051f923bf8a29a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:26.983120 9085 ts_tablet_manager.cc:1918] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:26.983191 9085 log.cc:1196] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/wal/wals/133ce4d0c83d48f39b8f733ed718c401
I20260924 23:16:26.983224 9067 tablet_bootstrap.cc:492] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913: Bootstrap starting.
I20260924 23:16:26.983441 9061 tablet_bootstrap.cc:492] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959: Bootstrap starting.
I20260924 23:16:26.983476 9085 ts_tablet_manager.cc:1939] T 133ce4d0c83d48f39b8f733ed718c401 P 9fbdf25c612649c9a6506b26fc3db913: Deleting consensus metadata
I20260924 23:16:26.983640 9067 tablet_bootstrap.cc:654] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:26.983852 9061 tablet_bootstrap.cc:654] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:26.984187 8537 catalog_manager.cc:4841] TS 9fbdf25c612649c9a6506b26fc3db913 (127.1.4.193:42525): tablet 133ce4d0c83d48f39b8f733ed718c401 (table test-table [id=ef0c7815d88a4c07bb0fc874bfe9176c]) successfully deleted
I20260924 23:16:26.981643 9086 ts_tablet_manager.cc:1905] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:26.984422 8810 tablet_service.cc:1467] Processing CreateTablet for tablet 0f83153bc55544cba051f923bf8a29a8 (DEFAULT_TABLE table=test-table [id=9acc228ece964de590309b79b9ac8ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:26.984548 8810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f83153bc55544cba051f923bf8a29a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:26.985615 9086 ts_tablet_manager.cc:1918] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:26.985690 9086 log.cc:1196] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/133ce4d0c83d48f39b8f733ed718c401
I20260924 23:16:26.985919 9062 tablet_bootstrap.cc:492] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776: Bootstrap starting.
I20260924 23:16:26.985957 9086 ts_tablet_manager.cc:1939] T 133ce4d0c83d48f39b8f733ed718c401 P 24665fd26a98463d864d9dcdb6321959: Deleting consensus metadata
I20260924 23:16:26.986289 9062 tablet_bootstrap.cc:654] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:26.986456 8534 catalog_manager.cc:4841] TS 24665fd26a98463d864d9dcdb6321959 (127.1.4.195:37963): tablet 133ce4d0c83d48f39b8f733ed718c401 (table test-table [id=ef0c7815d88a4c07bb0fc874bfe9176c]) successfully deleted
I20260924 23:16:26.987126 9067 tablet_bootstrap.cc:492] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913: No bootstrap required, opened a new log
I20260924 23:16:26.987183 9067 ts_tablet_manager.cc:1397] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 23:16:26.987330 9067 raft_consensus.cc:348] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:26.987432 9067 raft_consensus.cc:374] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:26.987459 9067 raft_consensus.cc:729] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbdf25c612649c9a6506b26fc3db913, State: Initialized, Role: FOLLOWER
I20260924 23:16:26.987499 9067 consensus_queue.cc:260] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:26.987581 9067 ts_tablet_manager.cc:1428] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:26.987686 9084 ts_tablet_manager.cc:1905] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:26.989100 9084 ts_tablet_manager.cc:1918] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:26.989168 9084 log.cc:1196] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/133ce4d0c83d48f39b8f733ed718c401
I20260924 23:16:26.989254 9062 tablet_bootstrap.cc:492] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776: No bootstrap required, opened a new log
I20260924 23:16:26.989297 9062 ts_tablet_manager.cc:1397] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 23:16:26.989390 9084 ts_tablet_manager.cc:1939] T 133ce4d0c83d48f39b8f733ed718c401 P a4aee7e753324811a3ad389c9e942776: Deleting consensus metadata
I20260924 23:16:26.989434 9062 raft_consensus.cc:348] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:26.989517 9062 raft_consensus.cc:374] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:26.989537 9062 raft_consensus.cc:729] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4aee7e753324811a3ad389c9e942776, State: Initialized, Role: FOLLOWER
I20260924 23:16:26.989578 9062 consensus_queue.cc:260] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:26.989669 9062 ts_tablet_manager.cc:1428] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:26.989758 8536 catalog_manager.cc:4841] TS a4aee7e753324811a3ad389c9e942776 (127.1.4.194:42881): tablet 133ce4d0c83d48f39b8f733ed718c401 (table test-table [id=ef0c7815d88a4c07bb0fc874bfe9176c]) successfully deleted
I20260924 23:16:26.989833 9061 tablet_bootstrap.cc:492] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959: No bootstrap required, opened a new log
I20260924 23:16:26.989868 9061 ts_tablet_manager.cc:1397] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260924 23:16:26.989964 9061 raft_consensus.cc:348] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:26.990010 9061 raft_consensus.cc:374] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:26.990031 9061 raft_consensus.cc:729] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24665fd26a98463d864d9dcdb6321959, State: Initialized, Role: FOLLOWER
I20260924 23:16:26.990065 9061 consensus_queue.cc:260] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:26.990126 9061 ts_tablet_manager.cc:1428] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:27.004405 9065 raft_consensus.cc:482] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:27.004504 9065 raft_consensus.cc:504] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.004834 9065 leader_election.cc:288] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fbdf25c612649c9a6506b26fc3db913 (127.1.4.193:42525), a4aee7e753324811a3ad389c9e942776 (127.1.4.194:42881)
I20260924 23:16:27.008242 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.005254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:56911 (local address 127.1.4.193:42525)
0924 23:16:27.005330 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:27.005332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:27.005340 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:27.005370 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:27.005371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:27.005379 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:27.005428 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:27.005607 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.006344 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.007464 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.007910 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.007988 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.007999 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:27.008135 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 23:16:27.008168 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 23:16:27.008265 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.004994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:42525 (local address 127.1.4.195:56911)
0924 23:16:27.005130 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:27.005141 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:27.005197 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:27.005458 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:27.005459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:27.005463 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:27.005525 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:27.005527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.006415 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.006416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:27.007321 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:27.007324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.007987 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.007988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:27.008063 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.008068 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:16:27.008122 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 23:16:27.008166 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":1,"thread_start_us":56,"threads_started":1}
I20260924 23:16:27.008571 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.005150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:49265 (local address 127.1.4.194:42881)
0924 23:16:27.005193 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:27.005195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:27.005206 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:16:27.005260 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:27.005261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:27.005272 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:27.005312 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:27.005520 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.006260 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.007706 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.008196 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.008256 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.008265 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:27.008454 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 23:16:27.008509 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:16:27.008556 8707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f83153bc55544cba051f923bf8a29a8" candidate_uuid: "24665fd26a98463d864d9dcdb6321959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fbdf25c612649c9a6506b26fc3db913" is_pre_election: true
I20260924 23:16:27.008698 8707 raft_consensus.cc:2454] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24665fd26a98463d864d9dcdb6321959 in term 0.
I20260924 23:16:27.008834 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.004997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:42881 (local address 127.1.4.195:49265)
0924 23:16:27.005142 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:27.005150 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:27.005197 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:27.005382 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:27.005383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:27.005385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:27.005457 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:27.005459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.006343 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.006345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:27.007621 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:27.007624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.008297 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.008299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:27.008368 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.008372 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:27.008705 (+ 333us) client_negotiation.cc:240] Negotiation successful
0924 23:16:27.008751 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":59,"threads_started":1}
I20260924 23:16:27.008885 8921 leader_election.cc:302] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24665fd26a98463d864d9dcdb6321959, 9fbdf25c612649c9a6506b26fc3db913; no voters:
I20260924 23:16:27.009028 9065 raft_consensus.cc:2781] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:27.009064 9065 raft_consensus.cc:482] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:27.009083 9065 raft_consensus.cc:3037] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:27.009142 8847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f83153bc55544cba051f923bf8a29a8" candidate_uuid: "24665fd26a98463d864d9dcdb6321959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4aee7e753324811a3ad389c9e942776" is_pre_election: true
I20260924 23:16:27.009222 8847 raft_consensus.cc:2454] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24665fd26a98463d864d9dcdb6321959 in term 0.
I20260924 23:16:27.009869 9065 raft_consensus.cc:504] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.010006 9065 leader_election.cc:288] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [CANDIDATE]: Term 1 election: Requested vote from peers 9fbdf25c612649c9a6506b26fc3db913 (127.1.4.193:42525), a4aee7e753324811a3ad389c9e942776 (127.1.4.194:42881)
I20260924 23:16:27.010169 8847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f83153bc55544cba051f923bf8a29a8" candidate_uuid: "24665fd26a98463d864d9dcdb6321959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4aee7e753324811a3ad389c9e942776"
I20260924 23:16:27.010187 8707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f83153bc55544cba051f923bf8a29a8" candidate_uuid: "24665fd26a98463d864d9dcdb6321959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fbdf25c612649c9a6506b26fc3db913"
I20260924 23:16:27.010247 8847 raft_consensus.cc:3037] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:27.010247 8707 raft_consensus.cc:3037] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:27.011099 8707 raft_consensus.cc:2454] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24665fd26a98463d864d9dcdb6321959 in term 1.
I20260924 23:16:27.011281 8921 leader_election.cc:302] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24665fd26a98463d864d9dcdb6321959, 9fbdf25c612649c9a6506b26fc3db913; no voters:
I20260924 23:16:27.011307 8847 raft_consensus.cc:2454] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24665fd26a98463d864d9dcdb6321959 in term 1.
I20260924 23:16:27.011391 9065 raft_consensus.cc:2781] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:27.011497 9065 raft_consensus.cc:686] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [term 1 LEADER]: Becoming Leader. State: Replica: 24665fd26a98463d864d9dcdb6321959, State: Running, Role: LEADER
I20260924 23:16:27.011577 9065 consensus_queue.cc:237] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.012219 8550 catalog_manager.cc:5495] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24665fd26a98463d864d9dcdb6321959 (127.1.4.195). New cstate: current_term: 1 leader_uuid: "24665fd26a98463d864d9dcdb6321959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } health_report { overall_health: HEALTHY } } }
Sep 24 23:16:27 dist-test-slave-s1hw krb5kdc[8505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291785, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.254@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM
I20260924 23:16:27.039532 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.032102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38399 (local address 127.0.0.1:56778)
0924 23:16:27.032281 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:27.032293 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:27.032376 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:27.032539 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:27.032540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:27.032920 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:27.032971 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:27.032974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.033604 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.033605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:27.034110 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:27.034112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.034409 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.034410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:27.034461 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.034975 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:27.034979 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:27.036451 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:27.038254 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:27.038261 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:27.038263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:27.038535 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:27.038758 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:27.038758 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:27.038758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:27.038827 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:27.039107 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:27.039114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:27.039168 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 23:16:27.039239 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 23:16:27.039385 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":104,"threads_started":1}
I20260924 23:16:27.039546 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.032169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56778 (local address 127.1.4.254:38399)
0924 23:16:27.032286 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:27.032288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:27.032371 (+ 83us) server_negotiation.cc:407] Connection header received
0924 23:16:27.032435 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:27.032436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:27.032446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:27.032523 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:27.033053 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.033536 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.034177 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.034352 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.034404 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.036540 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:27.036541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:27.036542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:27.036548 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:27.038166 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:27.038626 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:27.038627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:27.038627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:27.038689 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:27.038886 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:27.038887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:27.038887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:27.039009 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:27.039101 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:27.039286 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 23:16:27.039447 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260924 23:16:27.041545 8550 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:56778:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 23:16:27.041730 8550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:27.043139 8967 tablet_service.cc:1467] Processing CreateTablet for tablet 26ab00d9ce0a4dc49823d966df6dbe7e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3648e9b5f003408a95a8643619f1bb04]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:16:27.043128 8687 tablet_service.cc:1467] Processing CreateTablet for tablet 26ab00d9ce0a4dc49823d966df6dbe7e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3648e9b5f003408a95a8643619f1bb04]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:16:27.043140 8810 tablet_service.cc:1467] Processing CreateTablet for tablet 26ab00d9ce0a4dc49823d966df6dbe7e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3648e9b5f003408a95a8643619f1bb04]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:16:27.043287 8687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ab00d9ce0a4dc49823d966df6dbe7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:27.043287 8967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ab00d9ce0a4dc49823d966df6dbe7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:27.043288 8810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ab00d9ce0a4dc49823d966df6dbe7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:27.044763 9067 tablet_bootstrap.cc:492] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913: Bootstrap starting.
I20260924 23:16:27.044857 9061 tablet_bootstrap.cc:492] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959: Bootstrap starting.
I20260924 23:16:27.044869 9062 tablet_bootstrap.cc:492] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776: Bootstrap starting.
I20260924 23:16:27.045081 9067 tablet_bootstrap.cc:654] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:27.045217 9061 tablet_bootstrap.cc:654] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:27.045260 9062 tablet_bootstrap.cc:654] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:27.045804 9067 tablet_bootstrap.cc:492] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913: No bootstrap required, opened a new log
I20260924 23:16:27.045847 9067 ts_tablet_manager.cc:1397] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:27.045977 9062 tablet_bootstrap.cc:492] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776: No bootstrap required, opened a new log
I20260924 23:16:27.046018 9062 ts_tablet_manager.cc:1397] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:27.045985 9067 raft_consensus.cc:348] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.046048 9067 raft_consensus.cc:374] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:27.046061 9067 raft_consensus.cc:729] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbdf25c612649c9a6506b26fc3db913, State: Initialized, Role: FOLLOWER
I20260924 23:16:27.046095 9067 consensus_queue.cc:260] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.046180 9067 ts_tablet_manager.cc:1428] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:27.046152 9062 raft_consensus.cc:348] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.046214 9062 raft_consensus.cc:374] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:27.046232 9062 raft_consensus.cc:729] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4aee7e753324811a3ad389c9e942776, State: Initialized, Role: FOLLOWER
I20260924 23:16:27.046195 9072 tablet_replica.cc:439] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } }
I20260924 23:16:27.046255 9072 tablet_replica.cc:442] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:27.046270 9062 consensus_queue.cc:260] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.046355 9062 ts_tablet_manager.cc:1428] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:27.046491 9066 tablet_replica.cc:439] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } }
I20260924 23:16:27.046557 9066 tablet_replica.cc:442] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:27.046741 9061 tablet_bootstrap.cc:492] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959: No bootstrap required, opened a new log
I20260924 23:16:27.046794 9061 ts_tablet_manager.cc:1397] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:27.046933 9061 raft_consensus.cc:348] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.046999 9061 raft_consensus.cc:374] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:27.047019 9061 raft_consensus.cc:729] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24665fd26a98463d864d9dcdb6321959, State: Initialized, Role: FOLLOWER
I20260924 23:16:27.047086 9061 consensus_queue.cc:260] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.047175 9061 ts_tablet_manager.cc:1428] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:27.047308 9065 tablet_replica.cc:439] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } }
I20260924 23:16:27.047398 9065 tablet_replica.cc:442] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:27.212728 9072 raft_consensus.cc:482] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:27.212841 9072 raft_consensus.cc:504] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.212996 9072 leader_election.cc:288] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4aee7e753324811a3ad389c9e942776 (127.1.4.194:42881), 24665fd26a98463d864d9dcdb6321959 (127.1.4.195:37963)
I20260924 23:16:27.213266 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ab00d9ce0a4dc49823d966df6dbe7e" candidate_uuid: "9fbdf25c612649c9a6506b26fc3db913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24665fd26a98463d864d9dcdb6321959" is_pre_election: true
I20260924 23:16:27.213251 8847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ab00d9ce0a4dc49823d966df6dbe7e" candidate_uuid: "9fbdf25c612649c9a6506b26fc3db913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4aee7e753324811a3ad389c9e942776" is_pre_election: true
I20260924 23:16:27.213356 8987 raft_consensus.cc:2454] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbdf25c612649c9a6506b26fc3db913 in term 0.
I20260924 23:16:27.213366 8847 raft_consensus.cc:2454] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbdf25c612649c9a6506b26fc3db913 in term 0.
I20260924 23:16:27.213493 8638 leader_election.cc:302] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24665fd26a98463d864d9dcdb6321959, 9fbdf25c612649c9a6506b26fc3db913; no voters:
I20260924 23:16:27.213601 9072 raft_consensus.cc:2781] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:27.213647 9072 raft_consensus.cc:482] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:27.213667 9072 raft_consensus.cc:3037] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:27.214429 9072 raft_consensus.cc:504] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.214552 9072 leader_election.cc:288] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [CANDIDATE]: Term 1 election: Requested vote from peers a4aee7e753324811a3ad389c9e942776 (127.1.4.194:42881), 24665fd26a98463d864d9dcdb6321959 (127.1.4.195:37963)
I20260924 23:16:27.214701 8847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ab00d9ce0a4dc49823d966df6dbe7e" candidate_uuid: "9fbdf25c612649c9a6506b26fc3db913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4aee7e753324811a3ad389c9e942776"
I20260924 23:16:27.214725 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ab00d9ce0a4dc49823d966df6dbe7e" candidate_uuid: "9fbdf25c612649c9a6506b26fc3db913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24665fd26a98463d864d9dcdb6321959"
I20260924 23:16:27.214781 8847 raft_consensus.cc:3037] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:27.214787 8987 raft_consensus.cc:3037] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:27.215559 8847 raft_consensus.cc:2454] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbdf25c612649c9a6506b26fc3db913 in term 1.
I20260924 23:16:27.215572 8987 raft_consensus.cc:2454] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbdf25c612649c9a6506b26fc3db913 in term 1.
I20260924 23:16:27.215736 8638 leader_election.cc:302] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24665fd26a98463d864d9dcdb6321959, 9fbdf25c612649c9a6506b26fc3db913; no voters:
I20260924 23:16:27.215834 9072 raft_consensus.cc:2781] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:27.215896 9072 raft_consensus.cc:686] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [term 1 LEADER]: Becoming Leader. State: Replica: 9fbdf25c612649c9a6506b26fc3db913, State: Running, Role: LEADER
I20260924 23:16:27.215947 9072 consensus_queue.cc:237] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.216202 9069 tablet_replica.cc:439] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913: TxnStatusTablet state changed. Reason: New leader 9fbdf25c612649c9a6506b26fc3db913. Latest consensus state: current_term: 1 leader_uuid: "9fbdf25c612649c9a6506b26fc3db913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } }
I20260924 23:16:27.216282 9069 tablet_replica.cc:442] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:16:27.216470 9105 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 23:16:27.216583 9105 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 23:16:27.216563 8548 catalog_manager.cc:5495] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fbdf25c612649c9a6506b26fc3db913 (127.1.4.193). New cstate: current_term: 1 leader_uuid: "9fbdf25c612649c9a6506b26fc3db913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } health_report { overall_health: UNKNOWN } } }
I20260924 23:16:27.232532 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.229709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.1.4.193:42525)
0924 23:16:27.229757 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:27.229759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:27.229768 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:27.229922 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:27.229923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:27.229937 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:27.229994 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:27.230178 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.230705 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.231864 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.232116 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.232180 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.232336 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:27.232363 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:27.232396 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:27.232447 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 23:16:27.232464 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:16:27.232573 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.229634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:42525 (local address 127.0.0.1:47446)
0924 23:16:27.229697 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:27.229709 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:27.229757 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:27.230017 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:27.230018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:27.230019 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:16:27.230076 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:27.230079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.230787 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.230788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:27.231778 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:27.231781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.232190 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.232191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:27.232247 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.232263 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:16:27.232401 (+ 138us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:16:27.232406 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:16:27.232446 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 23:16:27.232490 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 23:16:27.233476 8987 raft_consensus.cc:1261] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959 [term 1 FOLLOWER]: Refusing update from remote peer 9fbdf25c612649c9a6506b26fc3db913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:27.233486 8847 raft_consensus.cc:1261] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Refusing update from remote peer 9fbdf25c612649c9a6506b26fc3db913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:27.233711 9072 consensus_queue.cc:1035] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:27.233829 9069 consensus_queue.cc:1035] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:27.234218 9065 tablet_replica.cc:439] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959: TxnStatusTablet state changed. Reason: New leader 9fbdf25c612649c9a6506b26fc3db913. Latest consensus state: current_term: 1 leader_uuid: "9fbdf25c612649c9a6506b26fc3db913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } }
I20260924 23:16:27.234303 9065 tablet_replica.cc:442] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:27.234339 9066 tablet_replica.cc:439] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776: TxnStatusTablet state changed. Reason: New leader 9fbdf25c612649c9a6506b26fc3db913. Latest consensus state: current_term: 1 leader_uuid: "9fbdf25c612649c9a6506b26fc3db913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } }
I20260924 23:16:27.234450 9066 tablet_replica.cc:442] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:27.234937 9069 tablet_replica.cc:439] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fbdf25c612649c9a6506b26fc3db913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } }
I20260924 23:16:27.235015 9069 tablet_replica.cc:442] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:16:27.235534 9065 tablet_replica.cc:439] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fbdf25c612649c9a6506b26fc3db913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } }
I20260924 23:16:27.235611 9065 tablet_replica.cc:442] T 26ab00d9ce0a4dc49823d966df6dbe7e P 24665fd26a98463d864d9dcdb6321959: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:27.236248 9072 tablet_replica.cc:439] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fbdf25c612649c9a6506b26fc3db913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } }
I20260924 23:16:27.236325 9072 tablet_replica.cc:442] T 26ab00d9ce0a4dc49823d966df6dbe7e P 9fbdf25c612649c9a6506b26fc3db913: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:16:27.236622 9066 tablet_replica.cc:439] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fbdf25c612649c9a6506b26fc3db913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } } }
I20260924 23:16:27.236714 9066 tablet_replica.cc:442] T 26ab00d9ce0a4dc49823d966df6dbe7e P a4aee7e753324811a3ad389c9e942776: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:16:27.236656 9076 mvcc.cc:205] Tried to move back new op lower bound from 7333035160506994688 to 7333035160436994048. Current Snapshot: MvccSnapshot[applied={T|T < 7333035160506994688}]
I20260924 23:16:27.239522 9059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.237290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.1.4.195:37963)
0924 23:16:27.237331 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:27.237333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:27.237342 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:27.237414 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:27.237414 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:27.237433 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:27.237493 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:27.237607 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.238078 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.238810 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.239011 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.239064 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.239205 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:27.239309 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:27.239399 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:27.239420 (+ 21us) server_negotiation.cc:299] Negotiation successful
0924 23:16:27.239443 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:16:27.244830 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.242190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:45931 (local address 127.1.4.193:42525)
0924 23:16:27.242235 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:27.242237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:27.242298 (+ 61us) server_negotiation.cc:407] Connection header received
0924 23:16:27.242366 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:27.242367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:27.242377 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:27.242425 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:27.242575 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.243046 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.244161 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.244347 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.244409 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.244610 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:27.244642 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:27.244695 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:27.244749 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 23:16:27.244767 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:16:27.244890 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.242109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:42525 (local address 127.1.4.195:45931)
0924 23:16:27.242244 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:27.242254 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:27.242313 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:27.242428 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:27.242429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:27.242430 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:16:27.242488 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:27.242489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.243118 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.243119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:27.244075 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:27.244078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.244433 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.244435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:27.244497 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.244513 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:16:27.244705 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:16:27.244706 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 23:16:27.244747 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 23:16:27.244790 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
I20260924 23:16:27.247450 8847 raft_consensus.cc:1261] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Refusing update from remote peer 24665fd26a98463d864d9dcdb6321959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:27.247519 8707 raft_consensus.cc:1261] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Refusing update from remote peer 24665fd26a98463d864d9dcdb6321959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:27.247786 9092 consensus_queue.cc:1035] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:27.247912 9065 consensus_queue.cc:1035] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:27.249625 9075 mvcc.cc:205] Tried to move back new op lower bound from 7333035160563671040 to 7333035159599992832. Current Snapshot: MvccSnapshot[applied={T|T < 7333035160563671040}]
I20260924 23:16:27.256278 9059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.253687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:38855 (local address 127.1.4.195:37963)
0924 23:16:27.253733 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:27.253735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:27.253803 (+ 68us) server_negotiation.cc:407] Connection header received
0924 23:16:27.253879 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:27.253879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:27.253888 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:27.253929 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:27.254036 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.254567 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.255677 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.255885 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.255948 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.256097 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:27.256122 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:27.256152 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:27.256208 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 23:16:27.256222 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 23:16:27.256327 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.253606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:37963 (local address 127.1.4.193:38855)
0924 23:16:27.253747 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:27.253758 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:27.253817 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:27.253933 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:27.253933 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:27.253935 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:16:27.253984 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:27.253986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.254631 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.254632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:27.255608 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:27.255611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.255953 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.255954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:27.256009 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.256025 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:16:27.256160 (+ 135us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:16:27.256161 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 23:16:27.256205 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:16:27.256246 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260924 23:16:27.265210 8548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:27.265307 8548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:27.265841 8548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d: Sending DeleteTablet for 3 replicas of tablet 0f83153bc55544cba051f923bf8a29a8
I20260924 23:16:27.266075 8687 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f83153bc55544cba051f923bf8a29a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:27 UTC) from {username='oryx', principal='oryx/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:47426
I20260924 23:16:27.266167 9085 tablet_replica.cc:330] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913: stopping tablet replica
I20260924 23:16:27.266180 8810 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f83153bc55544cba051f923bf8a29a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:27 UTC) from {username='oryx', principal='oryx/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:32796
I20260924 23:16:27.266211 9085 raft_consensus.cc:2229] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:27.266240 9085 raft_consensus.cc:2258] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:27.266270 9084 tablet_replica.cc:330] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776: stopping tablet replica
I20260924 23:16:27.266305 8967 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f83153bc55544cba051f923bf8a29a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:27 UTC) from {username='oryx', principal='oryx/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:57660
I20260924 23:16:27.266342 9084 raft_consensus.cc:2229] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:27.266379 9084 raft_consensus.cc:2258] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:27.266381 9086 tablet_replica.cc:330] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959: stopping tablet replica
I20260924 23:16:27.266414 9086 raft_consensus.cc:2229] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:16:27.266489 9086 raft_consensus.cc:2258] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:27.266690 9085 ts_tablet_manager.cc:1905] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:27.266726 9086 ts_tablet_manager.cc:1905] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:27.266844 9084 ts_tablet_manager.cc:1905] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:27.268428 9085 ts_tablet_manager.cc:1918] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:16:27.268460 9084 ts_tablet_manager.cc:1918] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:16:27.268489 9085 log.cc:1196] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/wal/wals/0f83153bc55544cba051f923bf8a29a8
I20260924 23:16:27.268519 9084 log.cc:1196] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/0f83153bc55544cba051f923bf8a29a8
I20260924 23:16:27.268683 9085 ts_tablet_manager.cc:1939] T 0f83153bc55544cba051f923bf8a29a8 P 9fbdf25c612649c9a6506b26fc3db913: Deleting consensus metadata
I20260924 23:16:27.268771 9084 ts_tablet_manager.cc:1939] T 0f83153bc55544cba051f923bf8a29a8 P a4aee7e753324811a3ad389c9e942776: Deleting consensus metadata
I20260924 23:16:27.269423 8536 catalog_manager.cc:4841] TS a4aee7e753324811a3ad389c9e942776 (127.1.4.194:42881): tablet 0f83153bc55544cba051f923bf8a29a8 (table test-table [id=9acc228ece964de590309b79b9ac8ff1]) successfully deleted
I20260924 23:16:27.269423 8537 catalog_manager.cc:4841] TS 9fbdf25c612649c9a6506b26fc3db913 (127.1.4.193:42525): tablet 0f83153bc55544cba051f923bf8a29a8 (table test-table [id=9acc228ece964de590309b79b9ac8ff1]) successfully deleted
I20260924 23:16:27.271111 9086 ts_tablet_manager.cc:1918] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:16:27.271167 9086 log.cc:1196] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/0f83153bc55544cba051f923bf8a29a8
I20260924 23:16:27.271342 9086 ts_tablet_manager.cc:1939] T 0f83153bc55544cba051f923bf8a29a8 P 24665fd26a98463d864d9dcdb6321959: Deleting consensus metadata
I20260924 23:16:27.271667 8534 catalog_manager.cc:4841] TS 24665fd26a98463d864d9dcdb6321959 (127.1.4.195:37963): tablet 0f83153bc55544cba051f923bf8a29a8 (table test-table [id=9acc228ece964de590309b79b9ac8ff1]) successfully deleted
Sep 24 23:16:27 dist-test-slave-s1hw krb5kdc[8505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 23:16:27 dist-test-slave-s1hw krb5kdc[8505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:27 dist-test-slave-s1hw krb5kdc[8505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 23:16:27.290495 9100 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:27.284209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.1.4.254:38399)
0924 23:16:27.284267 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:27.284269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:27.284366 (+ 97us) server_negotiation.cc:407] Connection header received
0924 23:16:27.284435 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:27.284436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:27.284455 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:27.284505 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:27.285048 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.285744 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.286863 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.287155 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.287197 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.290422 (+ 3225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56780: BlockingRecv error: recv got EOF from 127.0.0.1:56780 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 23:16:27 dist-test-slave-s1hw krb5kdc[8505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM
I20260924 23:16:27.298118 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.291258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56792 (local address 127.1.4.254:38399)
0924 23:16:27.291411 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:27.291413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:27.291713 (+ 300us) server_negotiation.cc:407] Connection header received
0924 23:16:27.291773 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:27.291774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:27.291784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:27.291825 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:27.292260 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.292745 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.293409 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.293637 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.293691 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.295884 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:27.295885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:27.295886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:27.295893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:27.297140 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:27.297442 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:27.297443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:27.297444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:27.297492 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:27.297654 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:27.297655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:27.297655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:27.297758 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:27.297847 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:27.297941 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 23:16:27.298026 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260924 23:16:27.299541 8548 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:16:27.299706 8548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:27.300896 8687 tablet_service.cc:1467] Processing CreateTablet for tablet baedb95c110f4be288f74870ba210e78 (DEFAULT_TABLE table=test-table [id=dec4cc4ccdd7436d831592a8e0e6606d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:27.301052 8687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baedb95c110f4be288f74870ba210e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:27.301968 9067 tablet_bootstrap.cc:492] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913: Bootstrap starting.
I20260924 23:16:27.302012 8967 tablet_service.cc:1467] Processing CreateTablet for tablet baedb95c110f4be288f74870ba210e78 (DEFAULT_TABLE table=test-table [id=dec4cc4ccdd7436d831592a8e0e6606d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:27.302114 8967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baedb95c110f4be288f74870ba210e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:27.302264 9067 tablet_bootstrap.cc:654] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:27.303154 9061 tablet_bootstrap.cc:492] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959: Bootstrap starting.
I20260924 23:16:27.303193 8810 tablet_service.cc:1467] Processing CreateTablet for tablet baedb95c110f4be288f74870ba210e78 (DEFAULT_TABLE table=test-table [id=dec4cc4ccdd7436d831592a8e0e6606d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:27.303303 8810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baedb95c110f4be288f74870ba210e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:27.303586 9061 tablet_bootstrap.cc:654] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:27.304283 9061 tablet_bootstrap.cc:492] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959: No bootstrap required, opened a new log
I20260924 23:16:27.304322 9061 ts_tablet_manager.cc:1397] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 23:16:27.304467 9061 raft_consensus.cc:348] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.304538 9061 raft_consensus.cc:374] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:27.304557 9061 raft_consensus.cc:729] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24665fd26a98463d864d9dcdb6321959, State: Initialized, Role: FOLLOWER
I20260924 23:16:27.304595 9061 consensus_queue.cc:260] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.304677 9061 ts_tablet_manager.cc:1428] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:27.304764 9062 tablet_bootstrap.cc:492] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776: Bootstrap starting.
I20260924 23:16:27.305094 9062 tablet_bootstrap.cc:654] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:27.305575 9067 tablet_bootstrap.cc:492] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913: No bootstrap required, opened a new log
I20260924 23:16:27.305624 9067 ts_tablet_manager.cc:1397] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 23:16:27.305765 9067 raft_consensus.cc:348] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.305835 9067 raft_consensus.cc:374] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:27.305855 9067 raft_consensus.cc:729] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbdf25c612649c9a6506b26fc3db913, State: Initialized, Role: FOLLOWER
I20260924 23:16:27.305894 9067 consensus_queue.cc:260] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.305987 9067 ts_tablet_manager.cc:1428] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:27.306514 9062 tablet_bootstrap.cc:492] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776: No bootstrap required, opened a new log
I20260924 23:16:27.306577 9062 ts_tablet_manager.cc:1397] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:27.306708 9062 raft_consensus.cc:348] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.306774 9062 raft_consensus.cc:374] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:27.306792 9062 raft_consensus.cc:729] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4aee7e753324811a3ad389c9e942776, State: Initialized, Role: FOLLOWER
I20260924 23:16:27.306833 9062 consensus_queue.cc:260] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.306914 9062 ts_tablet_manager.cc:1428] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:27.694939 9066 raft_consensus.cc:482] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:27.695052 9066 raft_consensus.cc:504] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.695452 9066 leader_election.cc:288] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fbdf25c612649c9a6506b26fc3db913 (127.1.4.193:42525), 24665fd26a98463d864d9dcdb6321959 (127.1.4.195:37963)
I20260924 23:16:27.699462 9059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.695792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:58495 (local address 127.1.4.195:37963)
0924 23:16:27.695846 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:27.695848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:27.695859 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:16:27.695916 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:27.695917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:27.695927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:27.695986 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:27.696197 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.696947 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.698545 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.699078 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.699161 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.699173 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:27.699335 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 23:16:27.699390 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 23:16:27.699482 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.695663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:37963 (local address 127.1.4.194:58495)
0924 23:16:27.695793 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:27.695800 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:27.695851 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:27.696048 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:27.696049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:27.696050 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:27.696100 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:27.696101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.697078 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.697079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:27.698433 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:27.698437 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.699173 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.699174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:27.699252 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.699257 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:16:27.699318 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 23:16:27.699382 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":40,"thread_start_us":57,"threads_started":1}
I20260924 23:16:27.699560 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.695747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:34537 (local address 127.1.4.193:42525)
0924 23:16:27.695814 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:27.695816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:27.695859 (+ 43us) server_negotiation.cc:407] Connection header received
0924 23:16:27.695932 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:27.695933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:27.695942 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:27.695994 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:27.696233 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.697006 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.698704 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.699230 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.699285 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.699292 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:27.699466 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 23:16:27.699504 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 23:16:27.699573 9137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.695655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:42525 (local address 127.1.4.194:34537)
0924 23:16:27.695779 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:27.695792 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:27.695866 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:27.696007 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:27.696007 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:27.696010 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:27.696092 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:27.696094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.697096 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.697097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:27.698619 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:27.698621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.699321 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.699321 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:27.699403 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.699406 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:27.699449 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 23:16:27.699509 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":43,"thread_start_us":69,"threads_started":1}
I20260924 23:16:27.699780 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baedb95c110f4be288f74870ba210e78" candidate_uuid: "a4aee7e753324811a3ad389c9e942776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24665fd26a98463d864d9dcdb6321959" is_pre_election: true
I20260924 23:16:27.699863 8987 raft_consensus.cc:2454] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4aee7e753324811a3ad389c9e942776 in term 0.
I20260924 23:16:27.700035 8778 leader_election.cc:302] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24665fd26a98463d864d9dcdb6321959, a4aee7e753324811a3ad389c9e942776; no voters:
I20260924 23:16:27.700110 8707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baedb95c110f4be288f74870ba210e78" candidate_uuid: "a4aee7e753324811a3ad389c9e942776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fbdf25c612649c9a6506b26fc3db913" is_pre_election: true
I20260924 23:16:27.700163 9066 raft_consensus.cc:2781] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:27.700201 8707 raft_consensus.cc:2454] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4aee7e753324811a3ad389c9e942776 in term 0.
I20260924 23:16:27.700201 9066 raft_consensus.cc:482] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:27.700225 9066 raft_consensus.cc:3037] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:27.700920 9066 raft_consensus.cc:504] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.701041 9066 leader_election.cc:288] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [CANDIDATE]: Term 1 election: Requested vote from peers 9fbdf25c612649c9a6506b26fc3db913 (127.1.4.193:42525), 24665fd26a98463d864d9dcdb6321959 (127.1.4.195:37963)
I20260924 23:16:27.701203 8707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baedb95c110f4be288f74870ba210e78" candidate_uuid: "a4aee7e753324811a3ad389c9e942776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fbdf25c612649c9a6506b26fc3db913"
I20260924 23:16:27.701228 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baedb95c110f4be288f74870ba210e78" candidate_uuid: "a4aee7e753324811a3ad389c9e942776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24665fd26a98463d864d9dcdb6321959"
I20260924 23:16:27.701303 8987 raft_consensus.cc:3037] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:27.701303 8707 raft_consensus.cc:3037] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:27.702148 8707 raft_consensus.cc:2454] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4aee7e753324811a3ad389c9e942776 in term 1.
I20260924 23:16:27.702148 8987 raft_consensus.cc:2454] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4aee7e753324811a3ad389c9e942776 in term 1.
I20260924 23:16:27.702364 8778 leader_election.cc:302] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24665fd26a98463d864d9dcdb6321959, a4aee7e753324811a3ad389c9e942776; no voters:
I20260924 23:16:27.702453 9066 raft_consensus.cc:2781] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:27.702581 9066 raft_consensus.cc:686] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [term 1 LEADER]: Becoming Leader. State: Replica: a4aee7e753324811a3ad389c9e942776, State: Running, Role: LEADER
I20260924 23:16:27.702651 9066 consensus_queue.cc:237] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.703437 8548 catalog_manager.cc:5495] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4aee7e753324811a3ad389c9e942776 (127.1.4.194). New cstate: current_term: 1 leader_uuid: "a4aee7e753324811a3ad389c9e942776" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } health_report { overall_health: UNKNOWN } } }
I20260924 23:16:27.749778 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.747299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32810 (local address 127.1.4.194:42881)
0924 23:16:27.747474 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:27.747476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:27.747482 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:16:27.747516 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:27.747517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:27.747536 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:27.747605 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:27.747742 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.748234 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.749008 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.749223 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.749279 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.749437 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:27.749553 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:27.749620 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:27.749658 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 23:16:27.749676 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":119,"threads_started":1}
I20260924 23:16:27.750684 8987 raft_consensus.cc:1261] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959 [term 1 FOLLOWER]: Refusing update from remote peer a4aee7e753324811a3ad389c9e942776: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:27.750703 8707 raft_consensus.cc:1261] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Refusing update from remote peer a4aee7e753324811a3ad389c9e942776: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:27.751000 9138 consensus_queue.cc:1035] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:27.751114 9066 consensus_queue.cc:1035] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:27.754016 8549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:27.754123 8549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:27.754711 8549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d: Sending DeleteTablet for 3 replicas of tablet baedb95c110f4be288f74870ba210e78
I20260924 23:16:27.754966 8967 tablet_service.cc:1514] Processing DeleteTablet for tablet baedb95c110f4be288f74870ba210e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:27 UTC) from {username='oryx', principal='oryx/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:57660
I20260924 23:16:27.754976 8687 tablet_service.cc:1514] Processing DeleteTablet for tablet baedb95c110f4be288f74870ba210e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:27 UTC) from {username='oryx', principal='oryx/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:47426
I20260924 23:16:27.755053 9086 tablet_replica.cc:330] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959: stopping tablet replica
I20260924 23:16:27.755057 9085 tablet_replica.cc:330] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913: stopping tablet replica
I20260924 23:16:27.755101 9086 raft_consensus.cc:2229] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:27.755101 9085 raft_consensus.cc:2229] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:27.755139 9085 raft_consensus.cc:2258] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:27.755142 9086 raft_consensus.cc:2258] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:27.755424 8810 tablet_service.cc:1514] Processing DeleteTablet for tablet baedb95c110f4be288f74870ba210e78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:27 UTC) from {username='oryx', principal='oryx/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:32796
I20260924 23:16:27.755535 9084 tablet_replica.cc:330] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776: stopping tablet replica
I20260924 23:16:27.755589 9084 raft_consensus.cc:2229] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:16:27.755666 9084 raft_consensus.cc:2258] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:27.755635 8549 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:16:27.755767 8549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:27.755843 9086 ts_tablet_manager.cc:1905] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:27.755957 9084 ts_tablet_manager.cc:1905] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:27.756093 9085 ts_tablet_manager.cc:1905] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:27.757586 9086 ts_tablet_manager.cc:1918] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:27.757637 9086 log.cc:1196] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/baedb95c110f4be288f74870ba210e78
I20260924 23:16:27.757665 9085 ts_tablet_manager.cc:1918] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:27.757725 9085 log.cc:1196] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-0/wal/wals/baedb95c110f4be288f74870ba210e78
I20260924 23:16:27.757829 9086 ts_tablet_manager.cc:1939] T baedb95c110f4be288f74870ba210e78 P 24665fd26a98463d864d9dcdb6321959: Deleting consensus metadata
I20260924 23:16:27.757937 9085 ts_tablet_manager.cc:1939] T baedb95c110f4be288f74870ba210e78 P 9fbdf25c612649c9a6506b26fc3db913: Deleting consensus metadata
I20260924 23:16:27.758109 8534 catalog_manager.cc:4841] TS 24665fd26a98463d864d9dcdb6321959 (127.1.4.195:37963): tablet baedb95c110f4be288f74870ba210e78 (table test-table [id=dec4cc4ccdd7436d831592a8e0e6606d]) successfully deleted
I20260924 23:16:27.758258 8967 tablet_service.cc:1467] Processing CreateTablet for tablet 12db1f2789ff47d4ad31a5bee5dd39e1 (DEFAULT_TABLE table=test-table [id=f9845b007b374167958f872b463cd7a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:27.758272 8537 catalog_manager.cc:4841] TS 9fbdf25c612649c9a6506b26fc3db913 (127.1.4.193:42525): tablet baedb95c110f4be288f74870ba210e78 (table test-table [id=dec4cc4ccdd7436d831592a8e0e6606d]) successfully deleted
I20260924 23:16:27.758355 8967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12db1f2789ff47d4ad31a5bee5dd39e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:27.758343 8687 tablet_service.cc:1467] Processing CreateTablet for tablet 12db1f2789ff47d4ad31a5bee5dd39e1 (DEFAULT_TABLE table=test-table [id=f9845b007b374167958f872b463cd7a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:27.758420 8687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12db1f2789ff47d4ad31a5bee5dd39e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:27.759186 9067 tablet_bootstrap.cc:492] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 9fbdf25c612649c9a6506b26fc3db913: Bootstrap starting.
I20260924 23:16:27.759496 9067 tablet_bootstrap.cc:654] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 9fbdf25c612649c9a6506b26fc3db913: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:27.759518 9061 tablet_bootstrap.cc:492] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959: Bootstrap starting.
I20260924 23:16:27.759780 9061 tablet_bootstrap.cc:654] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:27.759764 8810 tablet_service.cc:1467] Processing CreateTablet for tablet 12db1f2789ff47d4ad31a5bee5dd39e1 (DEFAULT_TABLE table=test-table [id=f9845b007b374167958f872b463cd7a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:27.759918 9084 ts_tablet_manager.cc:1918] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:27.759924 8810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12db1f2789ff47d4ad31a5bee5dd39e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:27.759966 9084 log.cc:1196] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/baedb95c110f4be288f74870ba210e78
I20260924 23:16:27.760138 9084 ts_tablet_manager.cc:1939] T baedb95c110f4be288f74870ba210e78 P a4aee7e753324811a3ad389c9e942776: Deleting consensus metadata
I20260924 23:16:27.760416 8536 catalog_manager.cc:4841] TS a4aee7e753324811a3ad389c9e942776 (127.1.4.194:42881): tablet baedb95c110f4be288f74870ba210e78 (table test-table [id=dec4cc4ccdd7436d831592a8e0e6606d]) successfully deleted
I20260924 23:16:27.760645 9067 tablet_bootstrap.cc:492] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 9fbdf25c612649c9a6506b26fc3db913: No bootstrap required, opened a new log
I20260924 23:16:27.760684 9067 ts_tablet_manager.cc:1397] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 9fbdf25c612649c9a6506b26fc3db913: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:16:27.760779 9067 raft_consensus.cc:348] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.760833 9067 raft_consensus.cc:374] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:27.760850 9067 raft_consensus.cc:729] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbdf25c612649c9a6506b26fc3db913, State: Initialized, Role: FOLLOWER
I20260924 23:16:27.760908 9067 consensus_queue.cc:260] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 9fbdf25c612649c9a6506b26fc3db913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.760993 9067 ts_tablet_manager.cc:1428] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 9fbdf25c612649c9a6506b26fc3db913: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:27.761168 9062 tablet_bootstrap.cc:492] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776: Bootstrap starting.
I20260924 23:16:27.761500 9062 tablet_bootstrap.cc:654] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:27.762041 9062 tablet_bootstrap.cc:492] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776: No bootstrap required, opened a new log
I20260924 23:16:27.762082 9062 ts_tablet_manager.cc:1397] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:27.762200 9062 raft_consensus.cc:348] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.762243 9062 raft_consensus.cc:374] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:27.762254 9062 raft_consensus.cc:729] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4aee7e753324811a3ad389c9e942776, State: Initialized, Role: FOLLOWER
I20260924 23:16:27.762283 9062 consensus_queue.cc:260] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.762365 9062 ts_tablet_manager.cc:1428] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:27.762759 9061 tablet_bootstrap.cc:492] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959: No bootstrap required, opened a new log
I20260924 23:16:27.762805 9061 ts_tablet_manager.cc:1397] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 23:16:27.762913 9061 raft_consensus.cc:348] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.762974 9061 raft_consensus.cc:374] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:27.762992 9061 raft_consensus.cc:729] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24665fd26a98463d864d9dcdb6321959, State: Initialized, Role: FOLLOWER
I20260924 23:16:27.763033 9061 consensus_queue.cc:260] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.763114 9061 ts_tablet_manager.cc:1428] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:27.852370 9119 raft_consensus.cc:482] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:27.852470 9119 raft_consensus.cc:504] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.852615 9119 leader_election.cc:288] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fbdf25c612649c9a6506b26fc3db913 (127.1.4.193:42525), a4aee7e753324811a3ad389c9e942776 (127.1.4.194:42881)
I20260924 23:16:27.852867 8707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12db1f2789ff47d4ad31a5bee5dd39e1" candidate_uuid: "24665fd26a98463d864d9dcdb6321959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fbdf25c612649c9a6506b26fc3db913" is_pre_election: true
I20260924 23:16:27.852932 8847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12db1f2789ff47d4ad31a5bee5dd39e1" candidate_uuid: "24665fd26a98463d864d9dcdb6321959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4aee7e753324811a3ad389c9e942776" is_pre_election: true
I20260924 23:16:27.852979 8707 raft_consensus.cc:2454] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24665fd26a98463d864d9dcdb6321959 in term 0.
I20260924 23:16:27.853013 8847 raft_consensus.cc:2454] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24665fd26a98463d864d9dcdb6321959 in term 0.
I20260924 23:16:27.853156 8921 leader_election.cc:302] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24665fd26a98463d864d9dcdb6321959, 9fbdf25c612649c9a6506b26fc3db913; no voters:
I20260924 23:16:27.853269 9119 raft_consensus.cc:2781] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:27.853302 9119 raft_consensus.cc:482] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:27.853322 9119 raft_consensus.cc:3037] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:27.854286 9119 raft_consensus.cc:504] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.854462 9119 leader_election.cc:288] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [CANDIDATE]: Term 1 election: Requested vote from peers 9fbdf25c612649c9a6506b26fc3db913 (127.1.4.193:42525), a4aee7e753324811a3ad389c9e942776 (127.1.4.194:42881)
I20260924 23:16:27.854621 8847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12db1f2789ff47d4ad31a5bee5dd39e1" candidate_uuid: "24665fd26a98463d864d9dcdb6321959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4aee7e753324811a3ad389c9e942776"
I20260924 23:16:27.854708 8847 raft_consensus.cc:3037] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:27.854708 8707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12db1f2789ff47d4ad31a5bee5dd39e1" candidate_uuid: "24665fd26a98463d864d9dcdb6321959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fbdf25c612649c9a6506b26fc3db913"
I20260924 23:16:27.854780 8707 raft_consensus.cc:3037] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 9fbdf25c612649c9a6506b26fc3db913 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:27.855446 8847 raft_consensus.cc:2454] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24665fd26a98463d864d9dcdb6321959 in term 1.
I20260924 23:16:27.855598 8707 raft_consensus.cc:2454] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24665fd26a98463d864d9dcdb6321959 in term 1.
I20260924 23:16:27.855615 8920 leader_election.cc:302] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24665fd26a98463d864d9dcdb6321959, a4aee7e753324811a3ad389c9e942776; no voters:
I20260924 23:16:27.855799 9119 raft_consensus.cc:2781] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:27.855919 9119 raft_consensus.cc:686] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [term 1 LEADER]: Becoming Leader. State: Replica: 24665fd26a98463d864d9dcdb6321959, State: Running, Role: LEADER
I20260924 23:16:27.856024 9119 consensus_queue.cc:237] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } }
I20260924 23:16:27.856596 8549 catalog_manager.cc:5495] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24665fd26a98463d864d9dcdb6321959 (127.1.4.195). New cstate: current_term: 1 leader_uuid: "24665fd26a98463d864d9dcdb6321959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24665fd26a98463d864d9dcdb6321959" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37963 } health_report { overall_health: HEALTHY } } }
I20260924 23:16:27.882485 9059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.880439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.1.4.195:37963)
0924 23:16:27.880494 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:27.880496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:27.880503 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:16:27.880523 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:27.880524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:27.880533 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:27.880572 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:27.880676 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.881169 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.881875 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:27.882070 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:27.882124 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:27.882249 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:27.882282 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:27.882323 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:27.882390 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 23:16:27.882410 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 23:16:27.884800 8847 raft_consensus.cc:1261] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Refusing update from remote peer 24665fd26a98463d864d9dcdb6321959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:27.885000 8707 raft_consensus.cc:1261] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Refusing update from remote peer 24665fd26a98463d864d9dcdb6321959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:27.884984 9148 consensus_queue.cc:1035] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4aee7e753324811a3ad389c9e942776" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 42881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:27.885180 9148 consensus_queue.cc:1035] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fbdf25c612649c9a6506b26fc3db913" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:27.894165 8550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:27.894310 8550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:27.894894 8550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1caf36f04a2149de8e3586732b16db2d: Sending DeleteTablet for 3 replicas of tablet 12db1f2789ff47d4ad31a5bee5dd39e1
I20260924 23:16:27.895147 8967 tablet_service.cc:1514] Processing DeleteTablet for tablet 12db1f2789ff47d4ad31a5bee5dd39e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:27 UTC) from {username='oryx', principal='oryx/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:57660
I20260924 23:16:27.895153 8687 tablet_service.cc:1514] Processing DeleteTablet for tablet 12db1f2789ff47d4ad31a5bee5dd39e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:27 UTC) from {username='oryx', principal='oryx/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:47426
I20260924 23:16:27.895177 8810 tablet_service.cc:1514] Processing DeleteTablet for tablet 12db1f2789ff47d4ad31a5bee5dd39e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:27 UTC) from {username='oryx', principal='oryx/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:32796
I20260924 23:16:27.895264 9086 tablet_replica.cc:330] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959: stopping tablet replica
I20260924 23:16:27.895357 9084 tablet_replica.cc:330] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776: stopping tablet replica
I20260924 23:16:27.895346 9085 tablet_replica.cc:330] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 9fbdf25c612649c9a6506b26fc3db913: stopping tablet replica
I20260924 23:16:27.895421 9084 raft_consensus.cc:2229] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:27.895455 9084 raft_consensus.cc:2258] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:27.895516 9085 raft_consensus.cc:2229] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:27.895572 9085 raft_consensus.cc:2258] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 9fbdf25c612649c9a6506b26fc3db913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:27.895644 9086 raft_consensus.cc:2229] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:16:27.895730 9086 raft_consensus.cc:2258] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:27.895984 9084 ts_tablet_manager.cc:1905] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:27.896373 9086 ts_tablet_manager.cc:1905] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:27.896471 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 8622
I20260924 23:16:27.898293 9084 ts_tablet_manager.cc:1918] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:16:27.898348 9084 log.cc:1196] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/12db1f2789ff47d4ad31a5bee5dd39e1
I20260924 23:16:27.898564 9084 ts_tablet_manager.cc:1939] T 12db1f2789ff47d4ad31a5bee5dd39e1 P a4aee7e753324811a3ad389c9e942776: Deleting consensus metadata
I20260924 23:16:27.898851 8536 catalog_manager.cc:4841] TS a4aee7e753324811a3ad389c9e942776 (127.1.4.194:42881): tablet 12db1f2789ff47d4ad31a5bee5dd39e1 (table test-table [id=f9845b007b374167958f872b463cd7a4]) successfully deleted
I20260924 23:16:27.899533 9086 ts_tablet_manager.cc:1918] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:16:27.899588 9086 log.cc:1196] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/12db1f2789ff47d4ad31a5bee5dd39e1
I20260924 23:16:27.899772 9086 ts_tablet_manager.cc:1939] T 12db1f2789ff47d4ad31a5bee5dd39e1 P 24665fd26a98463d864d9dcdb6321959: Deleting consensus metadata
I20260924 23:16:27.900133 8534 catalog_manager.cc:4841] TS 24665fd26a98463d864d9dcdb6321959 (127.1.4.195:37963): tablet 12db1f2789ff47d4ad31a5bee5dd39e1 (table test-table [id=f9845b007b374167958f872b463cd7a4]) successfully deleted
W20260924 23:16:27.903033 8537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.4.193:42525 (error 108)
I20260924 23:16:27.903249 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 8762
I20260924 23:16:27.904012 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:27.903396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:42525 (local address 127.0.0.1:47450)
0924 23:16:27.903938 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:27.903977 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:42525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":34,"threads_started":1}
W20260924 23:16:27.904100 8537 catalog_manager.cc:4584] TS 9fbdf25c612649c9a6506b26fc3db913 (127.1.4.193:42525): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12db1f2789ff47d4ad31a5bee5dd39e1: Network error: Client connection negotiation failed: client connection to 127.1.4.193:42525: connect: Connection refused (error 111)
I20260924 23:16:27.908876 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 8902
I20260924 23:16:27.913558 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 8518
2026-09-24T23:16:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3293 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:16:27 dist-test-slave-s1hw krb5kdc[9161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:16:27 dist-test-slave-s1hw krb5kdc[9161](info): set up 2 sockets
Sep 24 23:16:27 dist-test-slave-s1hw krb5kdc[9161](info): commencing operation
krb5kdc: starting...
W20260924 23:16:28.863718 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:16:28 dist-test-slave-s1hw krb5kdc[9161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:16:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:16:28.988221 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:35937
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:35937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:29.068253 9174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:29.068459 9174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:29.068480 9174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:29.069706 9174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:29.069756 9174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:29.069774 9174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:29.069789 9174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:29.069804 9174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:29.072109 9174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:35937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:35937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:29.072489 9174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:29.072737 9174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:29.078529 9179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:29.078529 9180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:29.078529 9182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:29.078841 9174 server_base.cc:1034] running on GCE node
I20260924 23:16:29.079109 9174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:29.079384 9174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:29.080541 9174 hybrid_clock.cc:648] HybridClock initialized: now 1790291789080528 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T23:16:29Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1172 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:16:29 dist-test-slave-s1hw krb5kdc[9187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:16:29 dist-test-slave-s1hw krb5kdc[9187](info): set up 2 sockets
Sep 24 23:16:29 dist-test-slave-s1hw krb5kdc[9187](info): commencing operation
krb5kdc: starting...
W20260924 23:16:30.032073 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:16:30 dist-test-slave-s1hw krb5kdc[9187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:16:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:16:30.151187 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:40219
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:40219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:30.230223 9200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:30.230439 9200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:30.230461 9200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:30.231658 9200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:30.231703 9200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:30.231717 9200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:30.231729 9200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:30.231738 9200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:30.233165 9200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:40219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:40219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:30.233354 9200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:30.233529 9200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:30.238211 9205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:30.238214 9206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:30.238214 9208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:30.238358 9200 server_base.cc:1034] running on GCE node
I20260924 23:16:30.238672 9200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:30.238945 9200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:30.240092 9200 hybrid_clock.cc:648] HybridClock initialized: now 1790291790240075 us; error 26 us; skew 500 ppm
Sep 24 23:16:30 dist-test-slave-s1hw krb5kdc[9187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:30.242429 9200 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:16:30.242867 9200 webserver.cc:460] Webserver started at http://127.1.4.254:38427/ using document root <none> and password file <none>
I20260924 23:16:30.243072 9200 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:30.243122 9200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:30.243244 9200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:30.244355 9200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "4ec8652f6bc845798d84cd79aaa8602c"
format_stamp: "Formatted at 2026-09-24 23:16:30 on dist-test-slave-s1hw"
server_key: "f71227a9320bcb1b4c150fca3900449e"
server_key_iv: "f9be889f28f1771e2fa84bb2252a439b"
server_key_version: "encryptionkey@0"
I20260924 23:16:30.244694 9200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "4ec8652f6bc845798d84cd79aaa8602c"
format_stamp: "Formatted at 2026-09-24 23:16:30 on dist-test-slave-s1hw"
server_key: "f71227a9320bcb1b4c150fca3900449e"
server_key_iv: "f9be889f28f1771e2fa84bb2252a439b"
server_key_version: "encryptionkey@0"
I20260924 23:16:30.245952 9200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:16:30.246721 9215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:30.246946 9200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:30.247030 9200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "4ec8652f6bc845798d84cd79aaa8602c"
format_stamp: "Formatted at 2026-09-24 23:16:30 on dist-test-slave-s1hw"
server_key: "f71227a9320bcb1b4c150fca3900449e"
server_key_iv: "f9be889f28f1771e2fa84bb2252a439b"
server_key_version: "encryptionkey@0"
I20260924 23:16:30.247090 9200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:30.261358 9200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:30.263545 9200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:30.263686 9200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:30.267220 9200 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:40219
I20260924 23:16:30.267261 9267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:40219 every 8 connection(s)
I20260924 23:16:30.267614 9200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:30.268463 9268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:30.270713 9268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c: Bootstrap starting.
I20260924 23:16:30.271251 9268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:30.271557 9268 log.cc:824] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:30.272255 9268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c: No bootstrap required, opened a new log
I20260924 23:16:30.273949 9268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec8652f6bc845798d84cd79aaa8602c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40219 } }
I20260924 23:16:30.274102 9268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:30.274124 9268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec8652f6bc845798d84cd79aaa8602c, State: Initialized, Role: FOLLOWER
I20260924 23:16:30.274250 9268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec8652f6bc845798d84cd79aaa8602c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40219 } }
I20260924 23:16:30.274326 9268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:30.274358 9268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:30.274400 9268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:30.274920 9268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec8652f6bc845798d84cd79aaa8602c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40219 } }
I20260924 23:16:30.275009 9268 leader_election.cc:302] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ec8652f6bc845798d84cd79aaa8602c; no voters:
I20260924 23:16:30.275151 9268 leader_election.cc:288] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:30.275205 9271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:30.275403 9268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:30.275583 9271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 1 LEADER]: Becoming Leader. State: Replica: 4ec8652f6bc845798d84cd79aaa8602c, State: Running, Role: LEADER
I20260924 23:16:30.275671 9271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec8652f6bc845798d84cd79aaa8602c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40219 } }
I20260924 23:16:30.275949 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 9200
I20260924 23:16:30.276048 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:30.276212 1043 external_mini_cluster.cc:1428] Setting key dd380d831821e131663f25e0132a6eb4
I20260924 23:16:30.276472 9273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ec8652f6bc845798d84cd79aaa8602c. Latest consensus state: current_term: 1 leader_uuid: "4ec8652f6bc845798d84cd79aaa8602c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec8652f6bc845798d84cd79aaa8602c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40219 } } }
I20260924 23:16:30.276567 9273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:30.276626 9272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ec8652f6bc845798d84cd79aaa8602c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec8652f6bc845798d84cd79aaa8602c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40219 } } }
I20260924 23:16:30.276839 9286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:30.277132 9272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:30.277663 9286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:30.279057 9286 catalog_manager.cc:1347] Generated new cluster ID: 0084311fadfe4f6da14b94f5df69db25
I20260924 23:16:30.279106 9286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:16:30 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:30.288268 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:30.276833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.1.4.254:40219)
0924 23:16:30.277189 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:30.277199 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:30.277237 (+ 38us) server_negotiation.cc:407] Connection header received
0924 23:16:30.277359 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:30.277367 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:30.277444 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:30.277603 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:30.278337 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:30.279731 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:30.280889 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:30.281516 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:30.281972 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:30.284471 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:30.284477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:30.284478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:30.284494 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:30.286476 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:30.286872 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:30.286876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:30.286876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:30.287015 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:30.287192 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:30.287192 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:30.287193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:30.287644 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:30.287812 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:30.287885 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 23:16:30.288033 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":66,"threads_started":1}
I20260924 23:16:30.304060 9286 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:30.304883 9286 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:30.316757 9286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c: Generated new TSK 0
I20260924 23:16:30.316987 9286 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:16:30.346719 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40219
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:30.426208 9294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:30.426419 9294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:30.426440 9294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:30.427650 9294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:30.427698 9294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:30.427724 9294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:16:30.429188 9294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:30.429430 9294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:30.429668 9294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:30.434954 9302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:30.434821 9299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:30.434830 9300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:30.434882 9294 server_base.cc:1034] running on GCE node
I20260924 23:16:30.435592 9294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:30.435818 9294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:30.436965 9294 hybrid_clock.cc:648] HybridClock initialized: now 1790291790436951 us; error 30 us; skew 500 ppm
Sep 24 23:16:30 dist-test-slave-s1hw krb5kdc[9187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:30.439432 9294 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:16:30.439877 9294 webserver.cc:460] Webserver started at http://127.1.4.193:40049/ using document root <none> and password file <none>
I20260924 23:16:30.440092 9294 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:30.440135 9294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:30.440228 9294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:30.441265 9294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "03090aefcfd6430f9b1b3ed082b55f13"
format_stamp: "Formatted at 2026-09-24 23:16:30 on dist-test-slave-s1hw"
server_key: "5c95b911551c9e8dfd105db3130aeb8a"
server_key_iv: "b19dbe4267755afeaef1a05beac4cf23"
server_key_version: "encryptionkey@0"
I20260924 23:16:30.441566 9294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "03090aefcfd6430f9b1b3ed082b55f13"
format_stamp: "Formatted at 2026-09-24 23:16:30 on dist-test-slave-s1hw"
server_key: "5c95b911551c9e8dfd105db3130aeb8a"
server_key_iv: "b19dbe4267755afeaef1a05beac4cf23"
server_key_version: "encryptionkey@0"
I20260924 23:16:30.442824 9294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:30.443728 9309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:30.443905 9294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:30.443984 9294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "03090aefcfd6430f9b1b3ed082b55f13"
format_stamp: "Formatted at 2026-09-24 23:16:30 on dist-test-slave-s1hw"
server_key: "5c95b911551c9e8dfd105db3130aeb8a"
server_key_iv: "b19dbe4267755afeaef1a05beac4cf23"
server_key_version: "encryptionkey@0"
I20260924 23:16:30.444058 9294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:30.462828 9294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:30.465121 9294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:30.465248 9294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:30.465515 9294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:30.465829 9294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:30.465862 9294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:30.465884 9294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:30.465895 9294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:30.471275 9294 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:35851
I20260924 23:16:30.471314 9422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:35851 every 8 connection(s)
I20260924 23:16:30.471685 9294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:16:30 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:30.480520 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:30.472345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:47413 (local address 127.1.4.254:40219)
0924 23:16:30.472400 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:30.472402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:30.472614 (+ 212us) server_negotiation.cc:407] Connection header received
0924 23:16:30.473077 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:30.473079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:30.473091 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:30.473146 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:30.474098 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:30.474609 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:30.475687 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:30.475889 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:30.475962 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:30.478041 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:30.478043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:30.478044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:30.478050 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:30.479300 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:30.479696 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:30.479697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:30.479698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:30.479746 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:30.479923 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:30.479924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:30.479924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:30.480021 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:30.480064 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:30.480347 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 23:16:30.480447 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 23:16:30.480576 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:30.472305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:47413)
0924 23:16:30.472495 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:30.472538 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:30.472999 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:30.473223 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:30.473227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:30.473785 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:30.473988 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:30.473993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:30.474685 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:30.474686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:30.475613 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:30.475616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:30.475957 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:30.475958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:30.476067 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:30.476615 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:30.476623 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:30.477952 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:30.479398 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:30.479406 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:30.479408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:30.479631 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:30.479790 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:30.479791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:30.479791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:30.479883 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:30.480068 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:30.480073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:30.480199 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 23:16:30.480283 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 23:16:30.480404 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260924 23:16:30.481153 9423 heartbeater.cc:347] Connected to a master server at 127.1.4.254:40219
I20260924 23:16:30.481240 9423 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:30.481389 9423 heartbeater.cc:511] Master 127.1.4.254:40219 requested a full tablet report, sending...
I20260924 23:16:30.481846 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 9294
I20260924 23:16:30.481966 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:16:30.481956 9232 ts_manager.cc:194] Registered new tserver with Master: 03090aefcfd6430f9b1b3ed082b55f13 (127.1.4.193:35851)
I20260924 23:16:30.482151 1043 external_mini_cluster.cc:1428] Setting key 76bf933b7f36b4a7d73a77993920c1a0
I20260924 23:16:30.482641 9232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:47413
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:16:30.507755 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40219
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:30.587673 9428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:30.587893 9428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:30.587916 9428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:30.589206 9428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:30.589258 9428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:30.589284 9428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:16:30.590739 9428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:30.590981 9428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:30.591195 9428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:16:30.596380 9428 server_base.cc:1034] running on GCE node
W20260924 23:16:30.596347 9433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:30.596334 9434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:30.596329 9436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:30.596764 9428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:30.597007 9428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:30.598158 9428 hybrid_clock.cc:648] HybridClock initialized: now 1790291790598146 us; error 29 us; skew 500 ppm
Sep 24 23:16:30 dist-test-slave-s1hw krb5kdc[9187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:30.600575 9428 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:16:30.601033 9428 webserver.cc:460] Webserver started at http://127.1.4.194:38751/ using document root <none> and password file <none>
I20260924 23:16:30.601214 9428 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:30.601253 9428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:30.601373 9428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:30.602422 9428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "b464c649ed1d462ebe5f47e1b044e34f"
format_stamp: "Formatted at 2026-09-24 23:16:30 on dist-test-slave-s1hw"
server_key: "527fccc9c5c0b560769a8e7766640aee"
server_key_iv: "1d258c1948938cbd7100782b4db81b27"
server_key_version: "encryptionkey@0"
I20260924 23:16:30.602713 9428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "b464c649ed1d462ebe5f47e1b044e34f"
format_stamp: "Formatted at 2026-09-24 23:16:30 on dist-test-slave-s1hw"
server_key: "527fccc9c5c0b560769a8e7766640aee"
server_key_iv: "1d258c1948938cbd7100782b4db81b27"
server_key_version: "encryptionkey@0"
I20260924 23:16:30.603924 9428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:30.604581 9443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:30.604722 9428 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:30.604796 9428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "b464c649ed1d462ebe5f47e1b044e34f"
format_stamp: "Formatted at 2026-09-24 23:16:30 on dist-test-slave-s1hw"
server_key: "527fccc9c5c0b560769a8e7766640aee"
server_key_iv: "1d258c1948938cbd7100782b4db81b27"
server_key_version: "encryptionkey@0"
I20260924 23:16:30.604858 9428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:30.618472 9428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:30.620563 9428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:30.620678 9428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:30.620898 9428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:30.621194 9428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:30.621227 9428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:30.621254 9428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:30.621272 9428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:30.626680 9428 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:44505
I20260924 23:16:30.626719 9556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:44505 every 8 connection(s)
I20260924 23:16:30.627032 9428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:16:30.632375 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 9428
I20260924 23:16:30.632498 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:16:30.632692 1043 external_mini_cluster.cc:1428] Setting key 7855e6e3efea9f4a5cb0a45d4c4e20c4
Sep 24 23:16:30 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:30.636332 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:30.627740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:56611 (local address 127.1.4.254:40219)
0924 23:16:30.627813 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:30.627815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:30.627967 (+ 152us) server_negotiation.cc:407] Connection header received
0924 23:16:30.628435 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:30.628436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:30.628445 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:30.628496 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:30.629446 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:30.629932 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:30.630993 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:30.631187 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:30.631251 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:30.633599 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:30.633600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:30.633601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:30.633608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:30.634873 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:30.635413 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:30.635414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:30.635415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:30.635477 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:30.635728 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:30.635729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:30.635730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:30.635843 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:30.635901 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:30.636162 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 23:16:30.636245 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 23:16:30.636498 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:30.627706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:56611)
0924 23:16:30.627847 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:30.627887 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:30.628357 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:30.628555 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:30.628558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:30.629168 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:30.629360 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:30.629365 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:30.630015 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:30.630016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:30.630861 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:30.630863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:30.631258 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:30.631259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:30.631414 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:30.631906 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:30.631914 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:30.633518 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:30.634961 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:30.634967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:30.634969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:30.635297 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:30.635545 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:30.635546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:30.635546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:30.635662 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:30.635897 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:30.635903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:30.636047 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 23:16:30.636134 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 23:16:30.636284 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260924 23:16:30.636971 9557 heartbeater.cc:347] Connected to a master server at 127.1.4.254:40219
I20260924 23:16:30.637060 9557 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:30.637224 9557 heartbeater.cc:511] Master 127.1.4.254:40219 requested a full tablet report, sending...
I20260924 23:16:30.637594 9232 ts_manager.cc:194] Registered new tserver with Master: b464c649ed1d462ebe5f47e1b044e34f (127.1.4.194:44505)
I20260924 23:16:30.638095 9232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:56611
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:16:30.656572 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40219
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:30.735301 9562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:30.735606 9562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:30.735635 9562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:30.736822 9562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:30.736857 9562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:30.736882 9562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:16:30.738276 9562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:30.738488 9562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:30.738670 9562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:30.743539 9570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:30.743564 9568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:30.743539 9567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:30.743741 9562 server_base.cc:1034] running on GCE node
I20260924 23:16:30.743956 9562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:30.744204 9562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:30.745354 9562 hybrid_clock.cc:648] HybridClock initialized: now 1790291790745334 us; error 26 us; skew 500 ppm
Sep 24 23:16:30 dist-test-slave-s1hw krb5kdc[9187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:30.747855 9562 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:16:30.748273 9562 webserver.cc:460] Webserver started at http://127.1.4.195:45489/ using document root <none> and password file <none>
I20260924 23:16:30.748481 9562 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:30.748524 9562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:30.748653 9562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:30.749764 9562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "38942980e2554492914af836974699f6"
format_stamp: "Formatted at 2026-09-24 23:16:30 on dist-test-slave-s1hw"
server_key: "fa9485b98a2723785a021cc689754971"
server_key_iv: "f5b3702a80539764b1a367ae040b73a9"
server_key_version: "encryptionkey@0"
I20260924 23:16:30.750085 9562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "38942980e2554492914af836974699f6"
format_stamp: "Formatted at 2026-09-24 23:16:30 on dist-test-slave-s1hw"
server_key: "fa9485b98a2723785a021cc689754971"
server_key_iv: "f5b3702a80539764b1a367ae040b73a9"
server_key_version: "encryptionkey@0"
I20260924 23:16:30.751406 9562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 23:16:30.752099 9577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:30.752233 9562 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:30.752310 9562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "38942980e2554492914af836974699f6"
format_stamp: "Formatted at 2026-09-24 23:16:30 on dist-test-slave-s1hw"
server_key: "fa9485b98a2723785a021cc689754971"
server_key_iv: "f5b3702a80539764b1a367ae040b73a9"
server_key_version: "encryptionkey@0"
I20260924 23:16:30.752373 9562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:30.771451 9562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:30.773281 9562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:30.773401 9562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:30.773636 9562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:30.773917 9562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:30.773950 9562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:30.773972 9562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:30.773985 9562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:30.779501 9562 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:33293
I20260924 23:16:30.779556 9690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:33293 every 8 connection(s)
I20260924 23:16:30.779855 9562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:16:30.780903 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 9562
I20260924 23:16:30.781042 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:16:30.781226 1043 external_mini_cluster.cc:1428] Setting key d0beaf93a00d0952702836eca35f635b
Sep 24 23:16:30 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:30.788736 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:30.780546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:60507 (local address 127.1.4.254:40219)
0924 23:16:30.780619 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:30.780621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:30.780750 (+ 129us) server_negotiation.cc:407] Connection header received
0924 23:16:30.781314 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:30.781316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:30.781328 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:30.781392 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:30.782461 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:30.782950 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:30.783935 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:30.784130 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:30.784181 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:30.786180 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:30.786182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:30.786182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:30.786189 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:30.787515 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:30.787853 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:30.787854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:30.787854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:30.787898 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:30.788119 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:30.788120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:30.788120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:30.788236 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:30.788291 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:30.788567 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 23:16:30.788645 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 23:16:30.788897 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:30.780506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:60507)
0924 23:16:30.780630 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:30.780667 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:30.781214 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:30.781477 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:30.781481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:30.782185 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:30.782382 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:30.782387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:30.783048 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:30.783048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:30.783876 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:30.783878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:30.784195 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:30.784196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:30.784309 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:30.784745 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:30.784753 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:30.786102 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:30.787581 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:30.787587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:30.787589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:30.787797 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:30.787939 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:30.787939 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:30.787940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:30.788073 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:30.788302 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:30.788309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:30.788441 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 23:16:30.788537 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 23:16:30.788696 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":65,"threads_started":1}
I20260924 23:16:30.789381 9691 heartbeater.cc:347] Connected to a master server at 127.1.4.254:40219
I20260924 23:16:30.789469 9691 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:30.789637 9691 heartbeater.cc:511] Master 127.1.4.254:40219 requested a full tablet report, sending...
I20260924 23:16:30.789966 9232 ts_manager.cc:194] Registered new tserver with Master: 38942980e2554492914af836974699f6 (127.1.4.195:33293)
I20260924 23:16:30.790302 9232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:60507
I20260924 23:16:30.792644 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.193.keytab.
Entry for principal oryx/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.193.keytab.
WARNING: no policy specified for oryx/127.1.4.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.4.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.194.keytab.
Entry for principal oryx/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.194.keytab.
WARNING: no policy specified for oryx/127.1.4.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.195.keytab.
Entry for principal oryx/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.195.keytab.
I20260924 23:16:30.862426 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 9294
I20260924 23:16:30.866326 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 9428
I20260924 23:16:30.870059 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 9562
I20260924 23:16:30.873787 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 9200
I20260924 23:16:30.878028 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:40219
--webserver_interface=127.1.4.254
--webserver_port=38427
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:40219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:30.957773 9700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:30.958005 9700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:30.958036 9700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:30.959200 9700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:30.959245 9700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:30.959263 9700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:30.959280 9700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:30.959293 9700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:30.960791 9700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:40219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:40219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=38427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:30.961043 9700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:30.961244 9700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:30.965780 9706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:30.965806 9705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:30.965806 9708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:30.966064 9700 server_base.cc:1034] running on GCE node
I20260924 23:16:30.966292 9700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:30.966566 9700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:30.967702 9700 hybrid_clock.cc:648] HybridClock initialized: now 1790291790967682 us; error 31 us; skew 500 ppm
Sep 24 23:16:30 dist-test-slave-s1hw krb5kdc[9187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:30.973985 9700 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:16:30.974453 9700 webserver.cc:460] Webserver started at http://127.1.4.254:38427/ using document root <none> and password file <none>
I20260924 23:16:30.974673 9700 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:30.974735 9700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:30.976109 9700 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:16:30.976716 9715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:30.976894 9700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:30.976989 9700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "4ec8652f6bc845798d84cd79aaa8602c"
format_stamp: "Formatted at 2026-09-24 23:16:30 on dist-test-slave-s1hw"
server_key: "f71227a9320bcb1b4c150fca3900449e"
server_key_iv: "f9be889f28f1771e2fa84bb2252a439b"
server_key_version: "encryptionkey@0"
I20260924 23:16:30.977340 9700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:31.015329 9700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:31.017405 9700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:31.017560 9700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:31.021319 9700 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:40219
I20260924 23:16:31.021431 9767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:40219 every 8 connection(s)
I20260924 23:16:31.021726 9700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:31.022841 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 9700
I20260924 23:16:31.023115 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:35851
--local_ip_for_outbound_sockets=127.1.4.193
--tserver_master_addrs=127.1.4.254:40219
--webserver_port=40049
--webserver_interface=127.1.4.193
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.4.193
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:16:31.023315 9768 sys_catalog.cc:263] Verifying existing consensus state
I20260924 23:16:31.024140 9768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c: Bootstrap starting.
I20260924 23:16:31.031911 9768 log.cc:824] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:31.033829 9768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:16:31.034127 9768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c: Bootstrap complete.
I20260924 23:16:31.036329 9768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec8652f6bc845798d84cd79aaa8602c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40219 } }
I20260924 23:16:31.036571 9768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec8652f6bc845798d84cd79aaa8602c, State: Initialized, Role: FOLLOWER
I20260924 23:16:31.036720 9768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec8652f6bc845798d84cd79aaa8602c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40219 } }
I20260924 23:16:31.036782 9768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:31.036811 9768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:31.036859 9768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 1 FOLLOWER]: Advancing to term 2
I20260924 23:16:31.037860 9768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec8652f6bc845798d84cd79aaa8602c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40219 } }
I20260924 23:16:31.037959 9768 leader_election.cc:302] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ec8652f6bc845798d84cd79aaa8602c; no voters:
I20260924 23:16:31.038118 9768 leader_election.cc:288] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 23:16:31.038169 9772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 23:16:31.038362 9772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [term 2 LEADER]: Becoming Leader. State: Replica: 4ec8652f6bc845798d84cd79aaa8602c, State: Running, Role: LEADER
I20260924 23:16:31.038386 9768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:31.038520 9772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec8652f6bc845798d84cd79aaa8602c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40219 } }
I20260924 23:16:31.038810 9774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ec8652f6bc845798d84cd79aaa8602c. Latest consensus state: current_term: 2 leader_uuid: "4ec8652f6bc845798d84cd79aaa8602c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec8652f6bc845798d84cd79aaa8602c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40219 } } }
I20260924 23:16:31.038825 9773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4ec8652f6bc845798d84cd79aaa8602c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ec8652f6bc845798d84cd79aaa8602c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40219 } } }
I20260924 23:16:31.038920 9773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:31.038877 9774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:31.039328 9778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:31.039841 9778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:31.039992 9778 catalog_manager.cc:1259] Loaded cluster ID: 0084311fadfe4f6da14b94f5df69db25
I20260924 23:16:31.040082 9778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:16:31.041018 9778 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:31.041134 9778 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4ec8652f6bc845798d84cd79aaa8602c: Loaded TSK: 0
I20260924 23:16:31.041266 9778 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 23:16:31.117666 9770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:31.117857 9770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:31.117880 9770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:31.119087 9770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:31.119128 9770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:31.119155 9770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:16:31.120625 9770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.193.keytab
--principal=oryx/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:35851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=40049
--tserver_master_addrs=127.1.4.254:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:31.120836 9770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:31.121079 9770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:31.125957 9794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:31.126081 9795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:31.125986 9797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:31.126297 9770 server_base.cc:1034] running on GCE node
I20260924 23:16:31.126505 9770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:31.126742 9770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:31.127885 9770 hybrid_clock.cc:648] HybridClock initialized: now 1790291791127871 us; error 31 us; skew 500 ppm
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291791, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:31.130503 9770 init.cc:377] Logged in from keytab as oryx/127.1.4.193@KRBTEST.COM (short username oryx)
I20260924 23:16:31.130980 9770 webserver.cc:460] Webserver started at http://127.1.4.193:40049/ using document root <none> and password file <none>
I20260924 23:16:31.131201 9770 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:31.131260 9770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:31.132728 9770 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:31.133555 9804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:31.133767 9770 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:31.133879 9770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "03090aefcfd6430f9b1b3ed082b55f13"
format_stamp: "Formatted at 2026-09-24 23:16:30 on dist-test-slave-s1hw"
server_key: "5c95b911551c9e8dfd105db3130aeb8a"
server_key_iv: "b19dbe4267755afeaef1a05beac4cf23"
server_key_version: "encryptionkey@0"
I20260924 23:16:31.134349 9770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:31.144142 9770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:31.146188 9770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:31.146319 9770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:31.146565 9770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:31.146850 9770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:31.146883 9770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:31.146911 9770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:31.146934 9770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:31.152609 9770 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:35851
I20260924 23:16:31.152658 9917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:35851 every 8 connection(s)
I20260924 23:16:31.153038 9770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:16:31.158080 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 9770
I20260924 23:16:31.158354 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:44505
--local_ip_for_outbound_sockets=127.1.4.194
--tserver_master_addrs=127.1.4.254:40219
--webserver_port=38751
--webserver_interface=127.1.4.194
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.4.194
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:31.161710 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:31.153710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:40691)
0924 23:16:31.153863 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:31.153906 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:31.154394 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:31.154806 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:31.154810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:31.155413 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:31.155613 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.155618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.156712 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.156714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.157492 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.157495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.158289 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.158291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.158437 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.159087 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:31.159097 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:31.161514 (+ 2417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
W20260924 23:16:31.161801 9921 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:31.153817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:40691 (local address 127.1.4.254:40219)
0924 23:16:31.153996 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:31.154004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:31.154045 (+ 41us) server_negotiation.cc:407] Connection header received
0924 23:16:31.154594 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:31.154601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:31.154644 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:31.154755 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:31.155718 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.156617 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.157570 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.158102 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.158207 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.161650 (+ 3443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:40691: BlockingRecv error: recv got EOF from 127.1.4.193:40691 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
W20260924 23:16:31.162024 9918 heartbeater.cc:644] Failed to heartbeat to 127.1.4.254:40219 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.4.254:40219: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:31.166571 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:31.162414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:50581)
0924 23:16:31.162537 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:31.162548 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:31.162594 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:31.162702 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:31.162703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:31.162822 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:31.162867 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.162868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.163502 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.163503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.164008 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.164010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.164307 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.164308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.164367 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.164827 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:31.164832 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:31.166506 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106}
W20260924 23:16:31.166642 9921 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:31.162499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:50581 (local address 127.1.4.254:40219)
0924 23:16:31.162543 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:31.162545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:31.162580 (+ 35us) server_negotiation.cc:407] Connection header received
0924 23:16:31.162638 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:31.162639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:31.162651 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:31.162695 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:31.162929 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.163425 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.164076 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.164254 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.164306 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.166593 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:50581: BlockingRecv error: recv got EOF from 127.1.4.193:50581 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:31.170562 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:31.166966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:49287)
0924 23:16:31.167047 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:31.167056 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:31.167097 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:31.167210 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:31.167211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:31.167324 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:31.167349 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.167350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.167941 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.167942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.168418 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.168419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.168706 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.168707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.168753 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.169112 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:31.169117 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:31.170500 (+ 1383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260924 23:16:31.170639 9921 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:31.167012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:49287 (local address 127.1.4.254:40219)
0924 23:16:31.167046 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:31.167048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:31.167084 (+ 36us) server_negotiation.cc:407] Connection header received
0924 23:16:31.167148 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:31.167148 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:31.167158 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:31.167201 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:31.167441 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.167870 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.168484 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.168654 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.168704 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.170589 (+ 1885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:49287: BlockingRecv error: recv got EOF from 127.1.4.193:49287 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 23:16:31.170894 9918 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 23:16:31.246985 9922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:31.247186 9922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:31.247207 9922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:31.248332 9922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:31.248370 9922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:31.248394 9922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:16:31.249742 9922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.194.keytab
--principal=oryx/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:44505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=38751
--tserver_master_addrs=127.1.4.254:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:31.249927 9922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:31.250095 9922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:31.254878 9927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:31.254952 9930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:31.255019 9922 server_base.cc:1034] running on GCE node
W20260924 23:16:31.254881 9928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:31.255268 9922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:31.255540 9922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:31.256676 9922 hybrid_clock.cc:648] HybridClock initialized: now 1790291791256662 us; error 32 us; skew 500 ppm
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291791, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:31.258977 9922 init.cc:377] Logged in from keytab as oryx/127.1.4.194@KRBTEST.COM (short username oryx)
I20260924 23:16:31.259416 9922 webserver.cc:460] Webserver started at http://127.1.4.194:38751/ using document root <none> and password file <none>
I20260924 23:16:31.259621 9922 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:31.259680 9922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:31.260963 9922 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:16:31.261538 9937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:31.261700 9922 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:16:31.261778 9922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "b464c649ed1d462ebe5f47e1b044e34f"
format_stamp: "Formatted at 2026-09-24 23:16:30 on dist-test-slave-s1hw"
server_key: "527fccc9c5c0b560769a8e7766640aee"
server_key_iv: "1d258c1948938cbd7100782b4db81b27"
server_key_version: "encryptionkey@0"
I20260924 23:16:31.262100 9922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:31.271247 9922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:31.273224 9922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:31.273355 9922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:31.273564 9922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:31.273850 9922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:31.273881 9922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:31.273909 9922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:31.273932 9922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:31.279354 9922 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:44505
I20260924 23:16:31.279441 10050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:44505 every 8 connection(s)
I20260924 23:16:31.279778 9922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:16:31.283797 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 9922
I20260924 23:16:31.284088 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:33293
--local_ip_for_outbound_sockets=127.1.4.195
--tserver_master_addrs=127.1.4.254:40219
--webserver_port=45489
--webserver_interface=127.1.4.195
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.4.195
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 23:16:31.287392 9921 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:31.280498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:53863 (local address 127.1.4.254:40219)
0924 23:16:31.280561 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:31.280563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:31.280758 (+ 195us) server_negotiation.cc:407] Connection header received
0924 23:16:31.281264 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:31.281265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:31.281279 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:31.281335 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:31.282216 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.282740 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.283762 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.283964 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.284039 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.287312 (+ 3273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:53863: BlockingRecv error: recv got EOF from 127.1.4.194:53863 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 23:16:31.287441 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:31.280518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:53863)
0924 23:16:31.280653 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:31.280686 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:31.281136 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:31.281390 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:31.281393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:31.281926 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:31.282118 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.282123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.282828 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.282829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.283681 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.283684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.284055 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.284056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.284227 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.284709 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:31.284719 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:31.287205 (+ 2486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
W20260924 23:16:31.287667 10051 heartbeater.cc:644] Failed to heartbeat to 127.1.4.254:40219 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.4.254:40219: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:31.292268 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:31.287896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:54595)
0924 23:16:31.287943 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:31.287954 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:31.287997 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:31.288155 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:31.288156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:31.288288 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:31.288346 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.288347 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.288974 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.288975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.289483 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.289485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.289815 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.289816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.289878 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.290293 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:31.290297 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:31.292198 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 23:16:31.292337 9921 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:31.287953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:54595 (local address 127.1.4.254:40219)
0924 23:16:31.288019 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:31.288021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:31.288042 (+ 21us) server_negotiation.cc:407] Connection header received
0924 23:16:31.288074 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:31.288075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:31.288089 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:31.288134 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:31.288416 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.288900 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.289556 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.289753 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.289813 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.292285 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:54595: BlockingRecv error: recv got EOF from 127.1.4.194:54595 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:31.296507 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:31.292500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:53363)
0924 23:16:31.292626 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:31.292635 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:31.292676 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:31.292786 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:31.292786 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:31.292898 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:31.292926 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.292928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.293527 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.293528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.294042 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.294043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.294356 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.294357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.294404 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.294774 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:31.294779 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:31.296437 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
W20260924 23:16:31.296618 9921 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:31.292553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:53363 (local address 127.1.4.254:40219)
0924 23:16:31.292589 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:31.292590 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:31.292677 (+ 87us) server_negotiation.cc:407] Connection header received
0924 23:16:31.292730 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:31.292731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:31.292740 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:31.292784 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:31.292986 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.293459 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.294118 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.294298 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.294351 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.296520 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:53363: BlockingRecv error: recv got EOF from 127.1.4.194:53363 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 23:16:31.296638 10051 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 23:16:31.365671 10054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:31.365865 10054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:31.365888 10054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:31.367020 10054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:31.367057 10054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:31.367082 10054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:16:31.368511 10054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/krb5kdc/oryx_127.1.4.195.keytab
--principal=oryx/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:33293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=45489
--tserver_master_addrs=127.1.4.254:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:31.368741 10054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:31.368957 10054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:31.373911 10060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:31.374037 10054 server_base.cc:1034] running on GCE node
W20260924 23:16:31.373926 10061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:31.374109 10063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:31.374285 10054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:31.374513 10054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:31.375655 10054 hybrid_clock.cc:648] HybridClock initialized: now 1790291791375641 us; error 27 us; skew 500 ppm
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291791, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:31.378132 10054 init.cc:377] Logged in from keytab as oryx/127.1.4.195@KRBTEST.COM (short username oryx)
I20260924 23:16:31.378583 10054 webserver.cc:460] Webserver started at http://127.1.4.195:45489/ using document root <none> and password file <none>
I20260924 23:16:31.378808 10054 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:31.378875 10054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:31.380326 10054 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:31.380983 10070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:31.381210 10054 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:31.381307 10054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "38942980e2554492914af836974699f6"
format_stamp: "Formatted at 2026-09-24 23:16:30 on dist-test-slave-s1hw"
server_key: "fa9485b98a2723785a021cc689754971"
server_key_iv: "f5b3702a80539764b1a367ae040b73a9"
server_key_version: "encryptionkey@0"
I20260924 23:16:31.381668 10054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:31.396884 10054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:31.398707 10054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:31.398826 10054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:31.399071 10054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:31.399392 10054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:31.399436 10054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:31.399466 10054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:31.399483 10054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:31.404846 10054 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:33293
I20260924 23:16:31.404901 10183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:33293 every 8 connection(s)
I20260924 23:16:31.405234 10054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:16:31.409152 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 10054
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:31.412832 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:31.405869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:59823)
0924 23:16:31.406051 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:31.406104 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:31.406635 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:31.406923 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:31.406927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:31.407529 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:31.407741 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.407746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.408394 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.408395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.409200 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.409202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.409662 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.409663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.409822 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.410393 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:31.410401 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:31.412678 (+ 2277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
W20260924 23:16:31.413069 10184 heartbeater.cc:644] Failed to heartbeat to 127.1.4.254:40219 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.4.254:40219: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
W20260924 23:16:31.413084 9921 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:31.405953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:59823 (local address 127.1.4.254:40219)
0924 23:16:31.406038 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:31.406040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:31.406191 (+ 151us) server_negotiation.cc:407] Connection header received
0924 23:16:31.406736 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:31.406738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:31.406749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:31.406805 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:31.407833 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.408309 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.409318 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.409547 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.409630 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.413028 (+ 3398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:59823: BlockingRecv error: recv got EOF from 127.1.4.195:59823 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 23:16:31.417099 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:31.409744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.1.4.254:40219)
0924 23:16:31.409877 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:31.409878 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:31.409885 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:16:31.409906 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:31.409906 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:31.409924 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:31.409977 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:31.410531 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.411131 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.412077 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.412375 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.412455 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.413892 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:31.413896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:31.413897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:31.413909 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:31.415624 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:31.416004 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:31.416007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:31.416007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:31.416095 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:31.416324 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:31.416325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:31.416325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:31.416600 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:31.416699 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:31.416888 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 23:16:31.417005 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":143,"threads_started":1}
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:31.418267 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:31.413350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:32959)
0924 23:16:31.413467 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:31.413479 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:31.413532 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:31.413640 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:31.413640 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:31.413775 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:31.413826 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.413828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.414677 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.414678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.415246 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.415248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.416024 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.416025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.416087 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.416442 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:31.416446 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:31.418221 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260924 23:16:31.418330 9921 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:31.413411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:32959 (local address 127.1.4.254:40219)
0924 23:16:31.413445 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:31.413447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:31.413517 (+ 70us) server_negotiation.cc:407] Connection header received
0924 23:16:31.413586 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:31.413586 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:31.413594 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:31.413633 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:31.413979 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.414596 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.415323 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.415618 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.415685 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.418288 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:32959: BlockingRecv error: recv got EOF from 127.1.4.195:32959 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:31 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:31.422004 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:31.418497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:34047)
0924 23:16:31.418528 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:31.418540 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:31.418583 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:31.418683 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:31.418684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:31.418783 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:31.418809 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.418810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.419435 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.419436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.419945 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:31.419947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.420234 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.420234 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:31.420266 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.420546 (+ 280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:31.420550 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:31.421932 (+ 1382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":14}
W20260924 23:16:31.422119 9921 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:31.418562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:34047 (local address 127.1.4.254:40219)
0924 23:16:31.418605 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:31.418607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:31.418612 (+ 5us) server_negotiation.cc:407] Connection header received
0924 23:16:31.418630 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:31.418630 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:31.418637 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:31.418672 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:31.418864 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.419332 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.420010 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:31.420183 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:31.420232 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:31.422073 (+ 1841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:34047: BlockingRecv error: recv got EOF from 127.1.4.195:34047 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260924 23:16:31.422286 10184 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 23:16:32 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:32 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:32.176050 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:32.171306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:57741)
0924 23:16:32.171501 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:32.171512 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:32.171582 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:32.171716 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:32.171716 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:32.171862 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:32.171908 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:32.171909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:32.172624 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:32.172624 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:32.173244 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:32.173248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:32.173664 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:32.173665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:32.173707 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:32.174177 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:32.174181 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:32.175960 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":130,"threads_started":1}
W20260924 23:16:32.176119 10190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:32.171405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:57741 (local address 127.1.4.254:40219)
0924 23:16:32.171529 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:32.171531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:32.171564 (+ 33us) server_negotiation.cc:407] Connection header received
0924 23:16:32.171633 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:32.171633 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:32.171644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:32.171709 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:32.172041 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:32.172552 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:32.173313 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:32.173595 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:32.173649 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:32.176060 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:57741: BlockingRecv error: recv got EOF from 127.1.4.193:57741 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
Sep 24 23:16:32 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:32 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:32.301329 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:32.297017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:48371)
0924 23:16:32.297164 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:32.297176 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:32.297246 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:32.297372 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:32.297373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:32.297544 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:32.297590 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:32.297592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:32.298279 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:32.298280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:32.298835 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:32.298837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:32.299167 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:32.299169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:32.299220 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:32.299666 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:32.299670 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:32.301241 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":58,"threads_started":1}
W20260924 23:16:32.301390 10190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:32.297117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:48371 (local address 127.1.4.254:40219)
0924 23:16:32.297180 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:32.297181 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:32.297238 (+ 57us) server_negotiation.cc:407] Connection header received
0924 23:16:32.297299 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:32.297300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:32.297312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:32.297357 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:32.297663 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:32.298198 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:32.298917 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:32.299109 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:32.299164 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:32.301339 (+ 2175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:48371: BlockingRecv error: recv got EOF from 127.1.4.194:48371 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 23:16:32 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:32 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:32.427552 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:32.422780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:49719)
0924 23:16:32.422905 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:32.422916 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:32.423007 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:32.423158 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:32.423159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:32.423353 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:32.423439 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:32.423442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:32.424218 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:32.424219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:32.424893 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:32.424896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:32.425280 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:32.425281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:32.425326 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:32.425772 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:32.425776 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:32.427482 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
W20260924 23:16:32.427646 10190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:32.422943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:49719 (local address 127.1.4.254:40219)
0924 23:16:32.422989 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:32.422991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:32.423002 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:16:32.423059 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:32.423060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:32.423072 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:32.423125 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:32.423541 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:32.424138 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:32.424955 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:32.425210 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:32.425257 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:32.427585 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:49719: BlockingRecv error: recv got EOF from 127.1.4.195:49719 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 23:16:33 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:33 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:33.181370 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:33.176536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:45559)
0924 23:16:33.176662 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:33.176672 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:33.176721 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:33.176851 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:33.176852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:33.177002 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:33.177046 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:33.177047 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:33.177728 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:33.177729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:33.178267 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:33.178270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:33.178691 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:33.178692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:33.178740 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:33.179214 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:33.179218 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:33.181303 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":56,"threads_started":1}
W20260924 23:16:33.181430 10194 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:33.176609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:45559 (local address 127.1.4.254:40219)
0924 23:16:33.176714 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:33.176716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:33.176725 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:33.176779 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:33.176780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:33.176790 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:33.176840 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:33.177105 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:33.177649 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:33.178382 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:33.178595 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:33.178664 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:33.181377 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:45559: BlockingRecv error: recv got EOF from 127.1.4.193:45559 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":50,"threads_started":1}
Sep 24 23:16:33 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:33 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:33.306545 10195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:33.301926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:33525)
0924 23:16:33.302118 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:33.302129 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:33.302187 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:33.302286 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:33.302287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:33.302436 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:33.302516 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:33.302518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:33.303176 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:33.303177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:33.303853 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:33.303856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:33.304196 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:33.304196 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:33.304234 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:33.304688 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:33.304694 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:33.306457 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":99,"threads_started":1}
W20260924 23:16:33.306614 10194 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:33.302022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:33525 (local address 127.1.4.254:40219)
0924 23:16:33.302072 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:33.302074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:33.302186 (+ 112us) server_negotiation.cc:407] Connection header received
0924 23:16:33.302225 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:33.302226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:33.302237 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:33.302283 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:33.302576 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:33.303094 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:33.303939 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:33.304133 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:33.304190 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:33.306561 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:33525: BlockingRecv error: recv got EOF from 127.1.4.194:33525 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 23:16:33 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:33 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:33.432896 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:33.428198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:53149)
0924 23:16:33.428369 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:33.428385 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:33.428461 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:33.428616 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:33.428617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:33.428781 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:33.428909 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:33.428912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:33.429593 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:33.429594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:33.430184 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:33.430186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:33.430553 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:33.430554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:33.430614 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:33.431072 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:33.431077 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:33.432813 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":70,"threads_started":1}
W20260924 23:16:33.433051 10194 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:33.428408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:53149 (local address 127.1.4.254:40219)
0924 23:16:33.428474 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:33.428475 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:33.428482 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:16:33.428508 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:33.428508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:33.428521 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:33.428569 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:33.429002 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:33.429519 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:33.430260 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:33.430469 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:33.430524 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:33.432979 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:53149: BlockingRecv error: recv got EOF from 127.1.4.195:53149 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 23:16:34 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:34 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:34.186607 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:34.181862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:52145)
0924 23:16:34.182057 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:34.182068 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:34.182126 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:34.182259 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:34.182260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:34.182412 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:34.182459 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:34.182460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:34.183157 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:34.183158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:34.183724 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:34.183726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:34.184159 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:34.184160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:34.184214 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:34.184625 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:34.184629 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:34.186548 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
W20260924 23:16:34.186707 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:34.182007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:52145 (local address 127.1.4.254:40219)
0924 23:16:34.182138 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:34.182139 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:34.182149 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:16:34.182173 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:34.182173 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:34.182183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:34.182243 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:34.182552 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:34.183088 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:34.183825 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:34.184069 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:34.184126 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:34.186643 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:52145: BlockingRecv error: recv got EOF from 127.1.4.193:52145 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
Sep 24 23:16:34 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:34 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:34.311909 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:34.307148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:41639)
0924 23:16:34.307281 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:34.307293 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:34.307356 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:34.307570 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:34.307571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:34.307731 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:34.307775 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:34.307776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:34.308420 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:34.308421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:34.309046 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:34.309049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:34.309400 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:34.309401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:34.309450 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:34.309893 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:34.309897 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:34.311814 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
W20260924 23:16:34.312070 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:34.307230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:41639 (local address 127.1.4.254:40219)
0924 23:16:34.307286 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:34.307288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:34.307355 (+ 67us) server_negotiation.cc:407] Connection header received
0924 23:16:34.307487 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:34.307488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:34.307501 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:34.307551 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:34.307848 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:34.308345 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:34.309134 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:34.309337 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:34.309391 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:34.312015 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:41639: BlockingRecv error: recv got EOF from 127.1.4.194:41639 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 23:16:34 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:34 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:34.438069 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:34.433539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:53105)
0924 23:16:34.433656 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:34.433668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:34.433730 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:34.433838 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:34.433839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:34.434036 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:34.434084 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:34.434085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:34.434710 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:34.434711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:34.435277 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:34.435279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:34.435634 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:34.435635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:34.435673 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:34.436079 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:34.436084 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:34.437997 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
W20260924 23:16:34.438221 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:34.433596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:53105 (local address 127.1.4.254:40219)
0924 23:16:34.433675 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:34.433677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:34.433737 (+ 60us) server_negotiation.cc:407] Connection header received
0924 23:16:34.433766 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:34.433767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:34.433779 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:34.433834 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:34.434152 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:34.434640 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:34.435387 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:34.435577 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:34.435630 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:34.438155 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:53105: BlockingRecv error: recv got EOF from 127.1.4.195:53105 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 23:16:35 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:35 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:35.191315 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:35.187083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:45093)
0924 23:16:35.187207 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:35.187217 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:35.187279 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:35.187420 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:35.187421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:35.187567 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:35.187611 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:35.187613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:35.188226 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:35.188227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:35.188746 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:35.188747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:35.189044 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:35.189045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:35.189085 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:35.189524 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:35.189528 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:35.191243 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
W20260924 23:16:35.191426 10202 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:35.187163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:45093 (local address 127.1.4.254:40219)
0924 23:16:35.187261 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:35.187263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:35.187271 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:35.187323 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:35.187324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:35.187334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:35.187412 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:35.187672 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:35.188156 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:35.188808 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:35.188988 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:35.189032 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:35.191333 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:45093: BlockingRecv error: recv got EOF from 127.1.4.193:45093 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":51,"threads_started":1}
Sep 24 23:16:35 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:35 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:35.316910 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:35.312420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:36361)
0924 23:16:35.312557 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:35.312568 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:35.312623 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:35.312780 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:35.312780 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:35.312939 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:35.313006 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:35.313007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:35.313652 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:35.313653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:35.314212 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:35.314214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:35.314605 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:35.314606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:35.314655 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:35.315076 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:35.315080 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:35.316828 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
W20260924 23:16:35.317075 10202 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:35.312463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:36361 (local address 127.1.4.254:40219)
0924 23:16:35.312519 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:35.312521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:35.312650 (+ 129us) server_negotiation.cc:407] Connection header received
0924 23:16:35.312673 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:35.312673 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:35.312685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:35.312734 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:35.313074 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:35.313560 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:35.314315 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:35.314518 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:35.314582 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:35.317018 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:36361: BlockingRecv error: recv got EOF from 127.1.4.194:36361 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 23:16:35 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:35 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:35.443446 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:35.438584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:38709)
0924 23:16:35.438728 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:35.438739 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:35.438808 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:35.439007 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:35.439008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:35.439164 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:35.439210 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:35.439212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:35.439870 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:35.439871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:35.440477 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:35.440479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:35.440845 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:35.440846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:35.440899 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:35.441409 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:35.441414 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:35.443330 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
W20260924 23:16:35.443483 10202 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:35.438670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:38709 (local address 127.1.4.254:40219)
0924 23:16:35.438730 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:35.438732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:35.438808 (+ 76us) server_negotiation.cc:407] Connection header received
0924 23:16:35.438855 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:35.438856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:35.438869 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:35.438938 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:35.439271 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:35.439789 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:35.440571 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:35.440771 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:35.440838 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:35.443433 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:38709: BlockingRecv error: recv got EOF from 127.1.4.195:38709 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 23:16:36 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:36 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:36.205082 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:36.191878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:52897)
0924 23:16:36.192038 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:36.192051 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:36.192112 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:36.192256 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:36.192257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:36.192464 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:36.192554 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:36.192558 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:36.193253 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:36.193254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:36.193814 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:36.193816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:36.194166 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:36.194167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:36.194203 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:36.203078 (+ 8875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:36.203085 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:36.205007 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
W20260924 23:16:36.205163 10206 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:36.191954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:52897 (local address 127.1.4.254:40219)
0924 23:16:36.192061 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:36.192063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:36.192095 (+ 32us) server_negotiation.cc:407] Connection header received
0924 23:16:36.192165 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:36.192166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:36.192178 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:36.192241 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:36.192626 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:36.193189 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:36.193880 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:36.194112 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:36.194162 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:36.205103 (+ 10941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:52897: BlockingRecv error: recv got EOF from 127.1.4.193:52897 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":46,"threads_started":1}
Sep 24 23:16:36 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:36 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:36.322321 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:36.317480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:50115)
0924 23:16:36.317618 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:36.317631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:36.317696 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:36.317853 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:36.317854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:36.318007 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:36.318055 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:36.318056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:36.318697 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:36.318699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:36.319291 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:36.319294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:36.319694 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:36.319696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:36.319749 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:36.320354 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:36.320359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:36.322259 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
W20260924 23:16:36.322432 10206 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:36.317633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:50115 (local address 127.1.4.254:40219)
0924 23:16:36.317721 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:36.317722 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:36.317730 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:36.317758 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:36.317759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:36.317770 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:36.317827 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:36.318118 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:36.318621 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:36.319392 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:36.319598 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:36.319651 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:36.322372 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:50115: BlockingRecv error: recv got EOF from 127.1.4.194:50115 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 23:16:36 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:36 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:36.448338 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:36.443905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:60849)
0924 23:16:36.444030 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:36.444041 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:36.444096 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:36.444272 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:36.444272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:36.444425 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:36.444477 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:36.444479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:36.445128 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:36.445129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:36.445666 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:36.445668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:36.446040 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:36.446041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:36.446104 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:36.446528 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:36.446532 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:36.448278 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":107,"threads_started":1}
W20260924 23:16:36.448454 10206 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:36.444018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:60849 (local address 127.1.4.254:40219)
0924 23:16:36.444083 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:36.444085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:36.444094 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:36.444188 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:36.444189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:36.444203 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:36.444261 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:36.444530 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:36.445071 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:36.445732 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:36.445912 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:36.446008 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:36.448393 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:60849: BlockingRecv error: recv got EOF from 127.1.4.195:60849 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 23:16:37 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:37 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:37.210250 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:37.205530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:47959)
0924 23:16:37.205663 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:37.205675 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:37.205739 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:37.205943 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:37.205945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:37.206111 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:37.206159 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:37.206161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:37.206852 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:37.206853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:37.207453 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:37.207455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:37.207815 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:37.207816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:37.207854 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:37.208262 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:37.208266 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:37.210191 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
W20260924 23:16:37.210403 10210 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:37.205628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:47959 (local address 127.1.4.254:40219)
0924 23:16:37.205768 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:37.205770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:37.205778 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:37.205799 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:37.205800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:37.205810 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:37.205897 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:37.206254 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:37.206761 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:37.207565 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:37.207758 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:37.207812 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:37.210327 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:47959: BlockingRecv error: recv got EOF from 127.1.4.193:47959 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
Sep 24 23:16:37 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:37 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:37.327598 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:37.322881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:54479)
0924 23:16:37.323040 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:37.323053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:37.323147 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:37.323313 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:37.323313 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:37.323565 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:37.323615 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:37.323617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:37.324441 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:37.324442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:37.324971 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:37.324973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:37.325316 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:37.325317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:37.325375 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:37.325816 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:37.325820 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:37.327535 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
W20260924 23:16:37.327731 10210 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:37.323029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:54479 (local address 127.1.4.254:40219)
0924 23:16:37.323098 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:37.323099 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:37.323127 (+ 28us) server_negotiation.cc:407] Connection header received
0924 23:16:37.323207 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:37.323208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:37.323226 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:37.323289 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:37.323697 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:37.324338 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:37.325050 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:37.325255 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:37.325310 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:37.327663 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:54479: BlockingRecv error: recv got EOF from 127.1.4.194:54479 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 23:16:37 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:37 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:37.453236 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:37.448847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:49161)
0924 23:16:37.449038 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:37.449050 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:37.449127 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:37.449258 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:37.449259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:37.449449 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:37.449494 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:37.449496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:37.450123 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:37.450124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:37.450688 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:37.450690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:37.451010 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:37.451011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:37.451062 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:37.451522 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:37.451527 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:37.453177 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":68,"threads_started":1}
W20260924 23:16:37.453436 10210 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:37.448943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:49161 (local address 127.1.4.254:40219)
0924 23:16:37.449008 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:37.449010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:37.449118 (+ 108us) server_negotiation.cc:407] Connection header received
0924 23:16:37.449180 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:37.449181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:37.449195 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:37.449242 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:37.449573 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:37.450051 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:37.450768 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:37.450953 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:37.451007 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:37.453374 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:49161: BlockingRecv error: recv got EOF from 127.1.4.195:49161 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 23:16:38 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:38 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:38.215626 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:38.210760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:47659)
0924 23:16:38.210886 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:38.210897 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:38.210950 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:38.211127 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:38.211128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:38.211292 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:38.211352 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:38.211354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:38.212094 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:38.212095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:38.212636 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:38.212638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:38.213039 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:38.213040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:38.213087 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:38.213633 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:38.213638 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:38.215567 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
W20260924 23:16:38.215754 10214 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:38.210875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:47659 (local address 127.1.4.254:40219)
0924 23:16:38.211002 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:38.211003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:38.211013 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:16:38.211036 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:38.211037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:38.211047 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:38.211111 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:38.211461 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:38.212009 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:38.212718 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:38.212915 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:38.213016 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:38.215683 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:47659: BlockingRecv error: recv got EOF from 127.1.4.193:47659 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
Sep 24 23:16:38 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:38 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:38.332701 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:38.328061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:35159)
0924 23:16:38.328254 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:38.328265 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:38.328327 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:38.328456 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:38.328457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:38.328612 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:38.328657 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:38.328659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:38.329339 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:38.329340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:38.329877 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:38.329878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:38.330305 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:38.330306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:38.330362 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:38.330821 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:38.330825 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:38.332641 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":127,"threads_started":1}
W20260924 23:16:38.332815 10214 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:38.328153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:35159 (local address 127.1.4.254:40219)
0924 23:16:38.328237 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:38.328239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:38.328363 (+ 124us) server_negotiation.cc:407] Connection header received
0924 23:16:38.328387 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:38.328388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:38.328399 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:38.328439 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:38.328750 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:38.329257 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:38.330024 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:38.330219 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:38.330281 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:38.332752 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:35159: BlockingRecv error: recv got EOF from 127.1.4.194:35159 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 23:16:38 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:38 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:38.458694 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:38.453707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:50369)
0924 23:16:38.453914 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:38.453926 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:38.453993 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:38.454116 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:38.454117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:38.454268 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:38.454314 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:38.454316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:38.455044 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:38.455045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:38.455653 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:38.455655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:38.456059 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:38.456060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:38.456110 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:38.456556 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:38.456560 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:38.458620 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
W20260924 23:16:38.458840 10214 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:38.453821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:50369 (local address 127.1.4.254:40219)
0924 23:16:38.453878 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:38.453880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:38.453986 (+ 106us) server_negotiation.cc:407] Connection header received
0924 23:16:38.454048 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:38.454049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:38.454062 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:38.454101 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:38.454382 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:38.454951 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:38.455752 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:38.455956 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:38.456026 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:38.458718 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:50369: BlockingRecv error: recv got EOF from 127.1.4.195:50369 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 23:16:39 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:39 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:39.220886 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:39.216148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:35149)
0924 23:16:39.216291 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:39.216302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:39.216361 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:39.216548 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:39.216549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:39.216704 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:39.216756 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:39.216758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:39.217532 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:39.217533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:39.218059 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:39.218061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:39.218415 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:39.218416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:39.218451 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:39.219050 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:39.219056 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:39.220799 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
W20260924 23:16:39.220968 10218 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:39.216261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:35149 (local address 127.1.4.254:40219)
0924 23:16:39.216365 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:39.216367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:39.216404 (+ 37us) server_negotiation.cc:407] Connection header received
0924 23:16:39.216428 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:39.216429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:39.216441 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:39.216492 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:39.216901 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:39.217425 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:39.218144 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:39.218351 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:39.218414 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:39.220904 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:35149: BlockingRecv error: recv got EOF from 127.1.4.193:35149 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":49,"threads_started":1}
Sep 24 23:16:39 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:39 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:39.338169 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:39.333229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:59575)
0924 23:16:39.333366 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:39.333380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:39.333448 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:39.333593 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:39.333594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:39.333761 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:39.333803 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:39.333805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:39.334718 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:39.334719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:39.335473 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:39.335476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:39.335853 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:39.335854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:39.335898 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:39.336307 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:39.336312 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:39.338109 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
W20260924 23:16:39.338248 10218 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:39.333355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:59575 (local address 127.1.4.254:40219)
0924 23:16:39.333448 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:39.333450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:39.333486 (+ 36us) server_negotiation.cc:407] Connection header received
0924 23:16:39.333513 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:39.333513 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:39.333525 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:39.333568 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:39.333970 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:39.334624 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:39.335554 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:39.335796 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:39.335845 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:39.338191 (+ 2346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:59575: BlockingRecv error: recv got EOF from 127.1.4.194:59575 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 23:16:39 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:39 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:39.463580 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:39.459202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:44519)
0924 23:16:39.459391 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:39.459404 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:39.459466 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:39.459579 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:39.459580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:39.459734 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:39.459780 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:39.459781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:39.460469 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:39.460471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:39.460990 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:39.460991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:39.461283 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:39.461284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:39.461320 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:39.461743 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:39.461747 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:39.463503 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
W20260924 23:16:39.463654 10218 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:39.459326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:44519 (local address 127.1.4.254:40219)
0924 23:16:39.459399 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:39.459401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:39.459452 (+ 51us) server_negotiation.cc:407] Connection header received
0924 23:16:39.459507 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:39.459508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:39.459519 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:39.459564 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:39.459838 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:39.460388 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:39.461054 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:39.461232 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:39.461279 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:39.463600 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:44519: BlockingRecv error: recv got EOF from 127.1.4.195:44519 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 23:16:40 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:40 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:40.226161 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:40.221463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:46835)
0924 23:16:40.221582 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:40.221594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:40.221646 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:40.221778 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:40.221778 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:40.221938 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:40.221993 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:40.221995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:40.222670 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:40.222670 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:40.223200 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:40.223203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:40.223708 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:40.223710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:40.223770 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:40.224205 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:40.224209 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:40.226082 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
W20260924 23:16:40.226261 10222 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:40.221526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:46835 (local address 127.1.4.254:40219)
0924 23:16:40.221643 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:40.221645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:40.221656 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:16:40.221705 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:40.221706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:40.221716 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:40.221766 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:40.222065 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:40.222604 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:40.223298 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:40.223585 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:40.223647 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:40.226196 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:46835: BlockingRecv error: recv got EOF from 127.1.4.193:46835 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
Sep 24 23:16:40 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:40 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:40.343482 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:40.338689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:48625)
0924 23:16:40.338827 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:40.338838 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:40.338990 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:40.339134 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:40.339135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:40.339300 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:40.339343 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:40.339345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:40.340052 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:40.340053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:40.340590 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:40.340592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:40.340993 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:40.340995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:40.341047 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:40.341519 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:40.341524 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:40.343417 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
W20260924 23:16:40.343631 10222 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:40.338746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:48625 (local address 127.1.4.254:40219)
0924 23:16:40.338815 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:40.338817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:40.338986 (+ 169us) server_negotiation.cc:407] Connection header received
0924 23:16:40.339059 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:40.339060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:40.339073 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:40.339120 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:40.339476 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:40.339990 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:40.340687 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:40.340911 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:40.340986 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:40.343564 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:48625: BlockingRecv error: recv got EOF from 127.1.4.194:48625 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 23:16:40 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:40 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:40.469329 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:40.464098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:38889)
0924 23:16:40.464248 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:40.464262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:40.464336 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:40.464483 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:40.464483 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:40.464680 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:40.464728 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:40.464730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:40.465526 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:40.465528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:40.466177 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:40.466180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:40.466511 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:40.466513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:40.466565 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:40.467074 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:40.467079 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:40.469232 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
W20260924 23:16:40.469447 10222 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:40.464206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:38889 (local address 127.1.4.254:40219)
0924 23:16:40.464264 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:40.464266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:40.464326 (+ 60us) server_negotiation.cc:407] Connection header received
0924 23:16:40.464407 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:40.464408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:40.464422 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:40.464476 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:40.464804 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:40.465451 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:40.466255 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:40.466451 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:40.466506 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:40.469344 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:38889: BlockingRecv error: recv got EOF from 127.1.4.195:38889 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 23:16:41 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:41 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:41.235810 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:41.226706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:48231)
0924 23:16:41.226851 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:41.226863 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:41.226919 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:41.227112 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:41.227113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:41.227266 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:41.227312 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:41.227314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:41.228018 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:41.228019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:41.228557 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:41.228559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:41.228866 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:41.228867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:41.228901 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:41.233870 (+ 4969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:41.233877 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:41.235749 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
W20260924 23:16:41.235953 10226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:41.226795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:48231 (local address 127.1.4.254:40219)
0924 23:16:41.226983 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:41.226984 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:41.226994 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:16:41.227017 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:41.227017 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:41.227030 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:41.227096 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:41.227435 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:41.227947 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:41.228622 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:41.228812 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:41.228865 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:41.235867 (+ 7002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:48231: BlockingRecv error: recv got EOF from 127.1.4.193:48231 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":118,"threads_started":1}
Sep 24 23:16:41 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:41 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:41.348675 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:41.344043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:48557)
0924 23:16:41.344168 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:41.344177 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:41.344232 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:41.344350 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:41.344351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:41.344535 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:41.344581 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:41.344583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:41.345212 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:41.345212 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:41.345755 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:41.345757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:41.346127 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:41.346129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:41.346179 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:41.346672 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:41.346676 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:41.348594 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":56,"threads_started":1}
W20260924 23:16:41.348753 10226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:41.344115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:48557 (local address 127.1.4.254:40219)
0924 23:16:41.344173 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:41.344175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:41.344226 (+ 51us) server_negotiation.cc:407] Connection header received
0924 23:16:41.344277 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:41.344277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:41.344288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:41.344341 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:41.344646 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:41.345136 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:41.345815 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:41.346042 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:41.346094 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:41.348701 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:48557: BlockingRecv error: recv got EOF from 127.1.4.194:48557 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 24 23:16:41 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:41 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:41.474615 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:41.469894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:54007)
0924 23:16:41.470051 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:41.470063 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:41.470123 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:41.470251 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:41.470252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:41.470427 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:41.470477 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:41.470479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:41.471156 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:41.471157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:41.471730 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:41.471733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:41.472075 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:41.472076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:41.472129 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:41.472635 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:41.472641 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:41.474540 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
W20260924 23:16:41.474694 10226 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:41.470059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:54007 (local address 127.1.4.254:40219)
0924 23:16:41.470132 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:41.470134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:41.470145 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:16:41.470174 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:41.470175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:41.470190 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:41.470238 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:41.470552 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:41.471070 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:41.471804 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:41.472011 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:41.472068 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:41.474642 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:54007: BlockingRecv error: recv got EOF from 127.1.4.195:54007 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 23:16:42 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:42 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:42.241662 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:42.236486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:34597)
0924 23:16:42.236652 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:42.236663 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:42.236742 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:42.236963 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:42.236963 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:42.237119 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:42.237165 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:42.237167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:42.237967 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:42.237968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:42.238602 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:42.238604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:42.239035 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:42.239037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:42.239096 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:42.239800 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:42.239805 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:42.241600 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
W20260924 23:16:42.241788 10230 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:42.236557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:34597 (local address 127.1.4.254:40219)
0924 23:16:42.236663 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:42.236665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:42.236785 (+ 120us) server_negotiation.cc:407] Connection header received
0924 23:16:42.236815 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:42.236816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:42.236828 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:42.236950 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:42.237255 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:42.237893 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:42.238679 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:42.238944 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:42.239019 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:42.241713 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:34597: BlockingRecv error: recv got EOF from 127.1.4.193:34597 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":53,"threads_started":1}
Sep 24 23:16:42 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:42 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:42.354058 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:42.349281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:48449)
0924 23:16:42.349473 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:42.349484 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:42.349555 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:42.349735 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:42.349737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:42.349943 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:42.349996 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:42.349999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:42.350641 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:42.350642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:42.351186 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:42.351189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:42.351621 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:42.351622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:42.351670 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:42.352129 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:42.352133 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:42.353992 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
W20260924 23:16:42.354171 10230 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:42.349373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:48449 (local address 127.1.4.254:40219)
0924 23:16:42.349424 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:42.349426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:42.349579 (+ 153us) server_negotiation.cc:407] Connection header received
0924 23:16:42.349612 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:42.349613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:42.349627 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:42.349688 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:42.350068 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:42.350571 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:42.351260 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:42.351524 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:42.351579 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:42.354107 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:48449: BlockingRecv error: recv got EOF from 127.1.4.194:48449 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 23:16:42 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:42 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:42.480014 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:42.475187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:51797)
0924 23:16:42.475327 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:42.475337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:42.475423 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:42.475561 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:42.475561 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:42.475718 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:42.475768 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:42.475770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:42.476497 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:42.476498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:42.477080 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:42.477082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:42.477413 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:42.477414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:42.477454 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:42.477851 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:42.477856 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:42.479936 (+ 2080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":60,"threads_started":1}
W20260924 23:16:42.480021 10230 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:42.475296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:51797 (local address 127.1.4.254:40219)
0924 23:16:42.475349 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:42.475351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:42.475406 (+ 55us) server_negotiation.cc:407] Connection header received
0924 23:16:42.475480 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:42.475480 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:42.475491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:42.475547 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:42.475918 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:42.476421 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:42.477155 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:42.477349 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:42.477402 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:42.479960 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:51797: BlockingRecv error: recv got EOF from 127.1.4.195:51797 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 23:16:43 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:43 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:43.246501 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:43.242131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:49611)
0924 23:16:43.242246 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:43.242257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:43.242309 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:43.242495 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:43.242497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:43.242652 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:43.242701 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:43.242703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:43.243414 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:43.243415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:43.243946 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:43.243948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:43.244280 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:43.244281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:43.244324 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:43.244800 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:43.244804 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:43.246441 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":51,"threads_started":1}
W20260924 23:16:43.246600 10234 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:43.242202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:49611 (local address 127.1.4.254:40219)
0924 23:16:43.242309 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:43.242311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:43.242347 (+ 36us) server_negotiation.cc:407] Connection header received
0924 23:16:43.242378 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:43.242378 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:43.242393 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:43.242442 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:43.242767 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:43.243302 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:43.244025 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:43.244221 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:43.244276 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:43.246532 (+ 2256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:49611: BlockingRecv error: recv got EOF from 127.1.4.193:49611 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":51,"threads_started":1}
Sep 24 23:16:43 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:43 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:43.359187 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:43.354576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:42533)
0924 23:16:43.354686 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:43.354698 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:43.354765 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:43.355063 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:43.355064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:43.355240 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:43.355284 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:43.355286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:43.355948 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:43.355950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:43.356491 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:43.356493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:43.356813 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:43.356814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:43.356875 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:43.357299 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:43.357304 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:43.359113 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
W20260924 23:16:43.359324 10234 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:43.354640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:42533 (local address 127.1.4.254:40219)
0924 23:16:43.354699 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:43.354701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:43.354768 (+ 67us) server_negotiation.cc:407] Connection header received
0924 23:16:43.354978 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:43.354979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:43.354993 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:43.355042 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:43.355382 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:43.355878 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:43.356566 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:43.356755 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:43.356812 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:43.359258 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:42533: BlockingRecv error: recv got EOF from 127.1.4.194:42533 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 23:16:43 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:43 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:43.484989 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:43.480574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:60203)
0924 23:16:43.480685 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:43.480697 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:43.480766 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:43.480887 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:43.480888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:43.481061 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:43.481116 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:43.481119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:43.481755 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:43.481756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:43.482290 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:43.482292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:43.482636 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:43.482636 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:43.482692 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:43.483127 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:43.483132 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:43.484892 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
W20260924 23:16:43.485038 10234 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:43.480653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:60203 (local address 127.1.4.254:40219)
0924 23:16:43.480700 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:43.480702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:43.480748 (+ 46us) server_negotiation.cc:407] Connection header received
0924 23:16:43.480816 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:43.480816 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:43.480831 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:43.480877 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:43.481193 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:43.481681 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:43.482370 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:43.482570 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:43.482625 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:43.484982 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:60203: BlockingRecv error: recv got EOF from 127.1.4.195:60203 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 23:16:44 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:44 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:44.251526 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:44.247039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:39911)
0924 23:16:44.247165 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:44.247176 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:44.247239 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:44.247434 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:44.247435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:44.247591 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:44.247639 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:44.247640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:44.248311 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:44.248312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:44.248857 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:44.248860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:44.249263 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:44.249264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:44.249325 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:44.249745 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:44.249750 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:44.251438 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
W20260924 23:16:44.251597 10238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:44.247147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:39911 (local address 127.1.4.254:40219)
0924 23:16:44.247268 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:44.247270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:44.247279 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:44.247311 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:44.247312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:44.247323 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:44.247418 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:44.247710 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:44.248244 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:44.248943 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:44.249188 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:44.249243 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:44.251536 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:39911: BlockingRecv error: recv got EOF from 127.1.4.193:39911 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":54,"threads_started":1}
Sep 24 23:16:44 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:44 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:44.364051 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:44.359684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:35589)
0924 23:16:44.359812 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:44.359824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:44.359870 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:44.360041 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:44.360042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:44.360194 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:44.360238 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:44.360240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:44.360979 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:44.360979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:44.361496 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:44.361498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:44.361899 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:44.361899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:44.361941 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:44.362358 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:44.362362 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:44.363979 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
W20260924 23:16:44.364148 10238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:44.359811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:35589 (local address 127.1.4.254:40219)
0924 23:16:44.359864 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:44.359866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:44.359875 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:44.359930 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:44.359931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:44.359988 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:44.360031 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:44.360329 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:44.360902 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:44.361578 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:44.361768 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:44.361896 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:44.364085 (+ 2189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:35589: BlockingRecv error: recv got EOF from 127.1.4.194:35589 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 23:16:44 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:44 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:44.489919 10240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:44.485463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:58645)
0924 23:16:44.485577 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:44.485589 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:44.485666 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:44.485802 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:44.485803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:44.485971 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:44.486016 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:44.486018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:44.486647 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:44.486648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:44.487228 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:44.487231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:44.487635 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:44.487636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:44.487687 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:44.488133 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:44.488137 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:44.489825 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
W20260924 23:16:44.490000 10238 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:44.485574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:58645 (local address 127.1.4.254:40219)
0924 23:16:44.485637 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:44.485639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:44.485651 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:16:44.485725 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:44.485726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:44.485740 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:44.485787 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:44.486089 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:44.486572 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:44.487323 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:44.487573 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:44.487631 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:44.489933 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:58645: BlockingRecv error: recv got EOF from 127.1.4.195:58645 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 23:16:45 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:45 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:45.256484 10241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:45.252005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:50415)
0924 23:16:45.252118 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:45.252129 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:45.252187 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:45.252325 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:45.252326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:45.252487 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:45.252531 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:45.252533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:45.253177 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:45.253178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:45.253737 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:45.253740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:45.254054 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:45.254055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:45.254096 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:45.254556 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:45.254560 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:45.256414 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
W20260924 23:16:45.256637 10242 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:45.252072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:50415 (local address 127.1.4.254:40219)
0924 23:16:45.252216 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:45.252217 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:45.252225 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:45.252245 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:45.252245 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:45.252256 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:45.252314 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:45.252601 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:45.253093 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:45.253802 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:45.253996 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:45.254041 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:45.256560 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:50415: BlockingRecv error: recv got EOF from 127.1.4.193:50415 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
Sep 24 23:16:45 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:45 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:45.369475 10243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:45.364650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:33517)
0924 23:16:45.364845 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:45.364859 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:45.364938 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:45.365079 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:45.365079 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:45.365248 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:45.365323 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:45.365325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:45.366081 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:45.366082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:45.366634 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:45.366637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:45.367021 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:45.367021 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:45.367070 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:45.367600 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:45.367605 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:45.369400 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
W20260924 23:16:45.369570 10242 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:45.364893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:33517 (local address 127.1.4.254:40219)
0924 23:16:45.364954 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:45.364956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:45.364967 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:16:45.365000 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:45.365001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:45.365012 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:45.365061 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:45.365423 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:45.366010 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:45.366730 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:45.366934 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:45.366998 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:45.369506 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:33517: BlockingRecv error: recv got EOF from 127.1.4.194:33517 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 23:16:45 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:45 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.195@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:45.495023 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:45.490471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.195:52407)
0924 23:16:45.490617 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:45.490628 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:45.490691 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:45.490822 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:45.490823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:45.491013 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:45.491057 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:45.491059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:45.491781 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:45.491782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:45.492386 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:45.492388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:45.492702 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:45.492703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:45.492753 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:45.493200 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:45.493205 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:45.494928 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
W20260924 23:16:45.495157 10242 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:45.490623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:52407 (local address 127.1.4.254:40219)
0924 23:16:45.490689 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:45.490691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:45.490701 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:16:45.490747 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:45.490748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:45.490762 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:45.490805 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:45.491137 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:45.491701 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:45.492456 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:45.492645 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:45.492699 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:45.495099 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.195:52407: BlockingRecv error: recv got EOF from 127.1.4.195:52407 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 23:16:46 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:46 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.193@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:46.267076 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:46.257013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.193:45751)
0924 23:16:46.257139 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:46.257149 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:46.257197 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:46.257330 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:46.257331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:46.257545 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:46.257593 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:46.257596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.258246 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.258246 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:46.258792 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:46.258794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.259177 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.259178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:46.259222 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:46.265067 (+ 5845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:46.265073 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:46.267016 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":55,"threads_started":1}
W20260924 23:16:46.267300 10246 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:46.257105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:45751 (local address 127.1.4.254:40219)
0924 23:16:46.257216 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:46.257217 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:46.257224 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:16:46.257245 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:46.257246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:46.257256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:46.257316 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:46.257655 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.258166 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.258896 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.259100 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.259164 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:46.267119 (+ 7955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.193:45751: BlockingRecv error: recv got EOF from 127.1.4.193:45751 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
Sep 24 23:16:46 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:16:46 dist-test-slave-s1hw krb5kdc[9187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.4.194@KRBTEST.COM for oryx/127.1.4.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:16:46.374588 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:46.370024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40219 (local address 127.1.4.194:49051)
0924 23:16:46.370173 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:46.370186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:46.370264 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:46.370420 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:46.370421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:46.370601 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:46.370646 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:46.370648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.371404 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.371431 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:46.371959 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:46.371961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.372279 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.372280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:46.372330 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:46.372750 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:46.372754 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:46.374506 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.4.254:40219: Server oryx/127.1.4.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
W20260924 23:16:46.374716 10246 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:46.370142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:49051 (local address 127.1.4.254:40219)
0924 23:16:46.370200 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:46.370203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:46.370245 (+ 42us) server_negotiation.cc:407] Connection header received
0924 23:16:46.370337 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:46.370338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:46.370351 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:46.370395 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:46.370747 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.371295 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.372038 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.372220 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.372275 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:46.374596 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.4.194:49051: BlockingRecv error: recv got EOF from 127.1.4.194:49051 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 23:16:46.417560 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 9770
I20260924 23:16:46.421908 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 9922
I20260924 23:16:46.425702 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 10054
I20260924 23:16:46.429562 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 9700
2026-09-24T23:16:46Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17342 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T23:16:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:46Z Disabled control of system clock
I20260924 23:16:46.455111 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:45587
--webserver_interface=127.1.4.254
--webserver_port=0
--builtin_ntp_servers=127.1.4.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:45587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 23:16:46.533396 10251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:46.533599 10251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:46.533617 10251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:46.534960 10251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:46.535008 10251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:46.535019 10251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:46.535030 10251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:46.535041 10251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:46.536453 10251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:45587
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:45587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:46.536674 10251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:46.536967 10251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:46.541725 10257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:46.541802 10251 server_base.cc:1034] running on GCE node
W20260924 23:16:46.541721 10256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:46.541721 10259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:46.542215 10251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:46.542514 10251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:46.543653 10251 hybrid_clock.cc:648] HybridClock initialized: now 1790291806543630 us; error 37 us; skew 500 ppm
I20260924 23:16:46.544860 10251 webserver.cc:460] Webserver started at http://127.1.4.254:44999/ using document root <none> and password file <none>
I20260924 23:16:46.545145 10251 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:46.545198 10251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:46.545316 10251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:46.546370 10251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "e733ab85f53b45fc9bf55273f644bb38"
format_stamp: "Formatted at 2026-09-24 23:16:46 on dist-test-slave-s1hw"
server_key: "575f49de6e38dcf44326daa0effcc766"
server_key_iv: "626186d5cb4edd364324adc0ddcc4d6a"
server_key_version: "encryptionkey@0"
I20260924 23:16:46.546703 10251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "e733ab85f53b45fc9bf55273f644bb38"
format_stamp: "Formatted at 2026-09-24 23:16:46 on dist-test-slave-s1hw"
server_key: "575f49de6e38dcf44326daa0effcc766"
server_key_iv: "626186d5cb4edd364324adc0ddcc4d6a"
server_key_version: "encryptionkey@0"
I20260924 23:16:46.548025 10251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:46.548724 10265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:46.548879 10251 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:46.548975 10251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "e733ab85f53b45fc9bf55273f644bb38"
format_stamp: "Formatted at 2026-09-24 23:16:46 on dist-test-slave-s1hw"
server_key: "575f49de6e38dcf44326daa0effcc766"
server_key_iv: "626186d5cb4edd364324adc0ddcc4d6a"
server_key_version: "encryptionkey@0"
I20260924 23:16:46.549042 10251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:46.571959 10251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:46.572252 10251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:46.572360 10251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:46.576085 10251 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:45587
I20260924 23:16:46.576128 10317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:45587 every 8 connection(s)
I20260924 23:16:46.576463 10251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:46.577152 10318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:46.579499 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 10251
I20260924 23:16:46.579573 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:46.579741 1043 external_mini_cluster.cc:1428] Setting key 7d7563f44412f6de690cf08ac5d6ed4c
I20260924 23:16:46.579854 10318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38: Bootstrap starting.
I20260924 23:16:46.580549 10318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:46.580849 10318 log.cc:824] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:46.581604 10318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38: No bootstrap required, opened a new log
I20260924 23:16:46.583266 10318 raft_consensus.cc:348] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e733ab85f53b45fc9bf55273f644bb38" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45587 } }
I20260924 23:16:46.583429 10318 raft_consensus.cc:374] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:46.583470 10318 raft_consensus.cc:729] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e733ab85f53b45fc9bf55273f644bb38, State: Initialized, Role: FOLLOWER
I20260924 23:16:46.583596 10318 consensus_queue.cc:260] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e733ab85f53b45fc9bf55273f644bb38" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45587 } }
I20260924 23:16:46.583681 10318 raft_consensus.cc:388] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:46.583712 10318 raft_consensus.cc:482] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:46.583756 10318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:46.584774 10318 raft_consensus.cc:504] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e733ab85f53b45fc9bf55273f644bb38" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45587 } }
I20260924 23:16:46.584884 10318 leader_election.cc:302] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e733ab85f53b45fc9bf55273f644bb38; no voters:
I20260924 23:16:46.585062 10318 leader_election.cc:288] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:46.585106 10323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:46.585248 10323 raft_consensus.cc:686] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [term 1 LEADER]: Becoming Leader. State: Replica: e733ab85f53b45fc9bf55273f644bb38, State: Running, Role: LEADER
I20260924 23:16:46.585345 10318 sys_catalog.cc:564] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:46.585381 10323 consensus_queue.cc:237] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e733ab85f53b45fc9bf55273f644bb38" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45587 } }
I20260924 23:16:46.585757 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:46.580252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.1.4.254:45587)
0924 23:16:46.580445 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:46.580463 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:46.580497 (+ 34us) server_negotiation.cc:407] Connection header received
0924 23:16:46.581030 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:46.581039 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:46.581121 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:46.581239 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:46.581464 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.582787 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.583655 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.584252 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.584383 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:46.585036 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:46.585040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:46.585041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:46.585240 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:46.585291 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:46.585307 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:46.585394 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:46.585474 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 23:16:46.585557 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260924 23:16:46.585842 10323 sys_catalog.cc:455] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e733ab85f53b45fc9bf55273f644bb38. Latest consensus state: current_term: 1 leader_uuid: "e733ab85f53b45fc9bf55273f644bb38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e733ab85f53b45fc9bf55273f644bb38" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45587 } } }
I20260924 23:16:46.585850 10325 sys_catalog.cc:455] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e733ab85f53b45fc9bf55273f644bb38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e733ab85f53b45fc9bf55273f644bb38" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45587 } } }
I20260924 23:16:46.585980 10323 sys_catalog.cc:458] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:46.585997 10325 sys_catalog.cc:458] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38 [sys.catalog]: This master's current role is: LEADER
W20260924 23:16:46.587306 10339 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:16:46.587348 10339 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:16:46.587409 10331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:46.587545 10331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:46.588898 10331 catalog_manager.cc:1347] Generated new cluster ID: 34e6949418474c3c87d515f311f12b2c
I20260924 23:16:46.588943 10331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:16:46.607817 10331 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:46.608366 10331 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:46.621944 10331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e733ab85f53b45fc9bf55273f644bb38: Generated new TSK 0
I20260924 23:16:46.622135 10331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:16:46.629470 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45587
--builtin_ntp_servers=127.1.4.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 23:16:46.710345 10342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:46.710565 10342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:46.710588 10342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:46.711757 10342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:46.711795 10342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:46.711818 10342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:16:46.713135 10342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:46.713351 10342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:46.713660 10342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:46.718358 10350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:46.718382 10348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:46.718420 10347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:46.718585 10342 server_base.cc:1034] running on GCE node
I20260924 23:16:46.718812 10342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:46.719044 10342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:46.720186 10342 hybrid_clock.cc:648] HybridClock initialized: now 1790291806720170 us; error 31 us; skew 500 ppm
I20260924 23:16:46.721423 10342 webserver.cc:460] Webserver started at http://127.1.4.193:46077/ using document root <none> and password file <none>
I20260924 23:16:46.721678 10342 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:46.721733 10342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:46.721951 10342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:46.723039 10342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "3806fefe3f4b4f0a9fe306bbb27544ce"
format_stamp: "Formatted at 2026-09-24 23:16:46 on dist-test-slave-s1hw"
server_key: "a919d3bb2664e74bb8731dfc38024998"
server_key_iv: "0ceb47332102d7de27562b4c57d3578a"
server_key_version: "encryptionkey@0"
I20260924 23:16:46.723435 10342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "3806fefe3f4b4f0a9fe306bbb27544ce"
format_stamp: "Formatted at 2026-09-24 23:16:46 on dist-test-slave-s1hw"
server_key: "a919d3bb2664e74bb8731dfc38024998"
server_key_iv: "0ceb47332102d7de27562b4c57d3578a"
server_key_version: "encryptionkey@0"
I20260924 23:16:46.724732 10342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:16:46.725468 10356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:46.725633 10342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:16:46.725708 10342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "3806fefe3f4b4f0a9fe306bbb27544ce"
format_stamp: "Formatted at 2026-09-24 23:16:46 on dist-test-slave-s1hw"
server_key: "a919d3bb2664e74bb8731dfc38024998"
server_key_iv: "0ceb47332102d7de27562b4c57d3578a"
server_key_version: "encryptionkey@0"
I20260924 23:16:46.725781 10342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:46.749410 10342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:46.749753 10342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:46.749897 10342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:46.750130 10342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:46.750454 10342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:46.750492 10342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:46.750525 10342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:46.750571 10342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:46.756691 10342 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:34019
I20260924 23:16:46.756747 10469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:34019 every 8 connection(s)
I20260924 23:16:46.757071 10342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:16:46.762442 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:46.757812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:53017 (local address 127.1.4.254:45587)
0924 23:16:46.757870 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:46.757872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:46.758132 (+ 260us) server_negotiation.cc:407] Connection header received
0924 23:16:46.758663 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:46.758664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:46.758679 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:46.758746 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:46.759471 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.760042 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.761147 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.761335 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.761397 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:46.762226 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:46.762227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:46.762227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:46.762231 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:46.762245 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:46.762249 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:46.762287 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:46.762358 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 23:16:46.762379 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 23:16:46.762662 10472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:46.757789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:45587 (local address 127.1.4.193:53017)
0924 23:16:46.757999 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:46.758043 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:46.758551 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:46.758809 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:46.758812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:46.758821 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:46.759320 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:46.759325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.760156 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.760157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:46.761051 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:46.761053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.761442 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.761443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:46.761566 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:46.762087 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:16:46.762087 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:46.762119 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:16:46.762145 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:46.762289 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:46.762291 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:46.762293 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:16:46.762355 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 23:16:46.762450 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260924 23:16:46.763231 10470 heartbeater.cc:347] Connected to a master server at 127.1.4.254:45587
I20260924 23:16:46.763325 10470 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:46.763535 10470 heartbeater.cc:511] Master 127.1.4.254:45587 requested a full tablet report, sending...
I20260924 23:16:46.764039 10282 ts_manager.cc:194] Registered new tserver with Master: 3806fefe3f4b4f0a9fe306bbb27544ce (127.1.4.193:34019)
I20260924 23:16:46.764315 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 10342
I20260924 23:16:46.764391 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:16:46.764573 1043 external_mini_cluster.cc:1428] Setting key 8333f9910c4ecd61925937d6122863b2
I20260924 23:16:46.764714 10282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.4.193:53017
I20260924 23:16:46.765547 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45587
--builtin_ntp_servers=127.1.4.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 23:16:46.844133 10473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:46.844344 10473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:46.844372 10473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:46.845513 10473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:46.845554 10473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:46.845585 10473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:16:46.846868 10473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:46.847105 10473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:46.847388 10473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:46.852512 10479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:46.852505 10478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:46.852684 10473 server_base.cc:1034] running on GCE node
W20260924 23:16:46.852505 10481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:46.852988 10473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:46.853235 10473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:46.854410 10473 hybrid_clock.cc:648] HybridClock initialized: now 1790291806854393 us; error 28 us; skew 500 ppm
I20260924 23:16:46.855693 10473 webserver.cc:460] Webserver started at http://127.1.4.194:33521/ using document root <none> and password file <none>
I20260924 23:16:46.856002 10473 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:46.856056 10473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:46.856180 10473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:46.857240 10473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "e4991b3dd4ac4c6b99dc75c37899984e"
format_stamp: "Formatted at 2026-09-24 23:16:46 on dist-test-slave-s1hw"
server_key: "aef91d329f7f7a64c7ea55f8aec3a205"
server_key_iv: "44f3a90f050f52440843c8c704ae1d70"
server_key_version: "encryptionkey@0"
I20260924 23:16:46.857568 10473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "e4991b3dd4ac4c6b99dc75c37899984e"
format_stamp: "Formatted at 2026-09-24 23:16:46 on dist-test-slave-s1hw"
server_key: "aef91d329f7f7a64c7ea55f8aec3a205"
server_key_iv: "44f3a90f050f52440843c8c704ae1d70"
server_key_version: "encryptionkey@0"
I20260924 23:16:46.858906 10473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:16:46.859740 10487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:46.859894 10473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:46.859979 10473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "e4991b3dd4ac4c6b99dc75c37899984e"
format_stamp: "Formatted at 2026-09-24 23:16:46 on dist-test-slave-s1hw"
server_key: "aef91d329f7f7a64c7ea55f8aec3a205"
server_key_iv: "44f3a90f050f52440843c8c704ae1d70"
server_key_version: "encryptionkey@0"
I20260924 23:16:46.860046 10473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:46.870179 10473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:46.870499 10473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:46.870656 10473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:46.870854 10473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:46.871178 10473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:46.871212 10473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:46.871239 10473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:46.871265 10473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:46.876652 10473 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:39595
I20260924 23:16:46.876713 10600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:39595 every 8 connection(s)
I20260924 23:16:46.877050 10473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:16:46.879822 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 10473
I20260924 23:16:46.879906 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:16:46.880100 1043 external_mini_cluster.cc:1428] Setting key 84d33718b555504eedc07fd284e9882f
I20260924 23:16:46.881043 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45587
--builtin_ntp_servers=127.1.4.212:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 23:16:46.882848 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:46.877728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:36703 (local address 127.1.4.254:45587)
0924 23:16:46.877784 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:46.877786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:46.878038 (+ 252us) server_negotiation.cc:407] Connection header received
0924 23:16:46.878589 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:46.878590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:46.878603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:46.878659 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:46.879345 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.879901 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.881292 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.881507 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.881563 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:46.882594 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:46.882596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:46.882596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:46.882601 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:46.882619 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:46.882624 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:46.882676 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:46.882740 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 23:16:46.882768 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:16:46.883074 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:46.877699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:45587 (local address 127.1.4.194:36703)
0924 23:16:46.877892 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:46.877937 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:46.878462 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:46.878736 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:46.878740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:46.878750 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:46.879247 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:46.879251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.880022 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.880023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:46.881219 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:46.881222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:46.881584 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:46.881585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:46.881736 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:46.882431 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:16:46.882431 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:46.882473 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:16:46.882502 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:46.882687 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:46.882690 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:46.882693 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:46.882739 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 23:16:46.882832 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260924 23:16:46.883582 10601 heartbeater.cc:347] Connected to a master server at 127.1.4.254:45587
I20260924 23:16:46.883673 10601 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:46.883826 10601 heartbeater.cc:511] Master 127.1.4.254:45587 requested a full tablet report, sending...
I20260924 23:16:46.884162 10281 ts_manager.cc:194] Registered new tserver with Master: e4991b3dd4ac4c6b99dc75c37899984e (127.1.4.194:39595)
I20260924 23:16:46.884644 10281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.4.194:36703
W20260924 23:16:46.962692 10604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:46.962929 10604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:46.962952 10604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:46.964242 10604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:46.964284 10604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:46.964310 10604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:16:46.965632 10604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:46.965842 10604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:46.966109 10604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:46.970947 10609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:46.971016 10610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:46.971073 10612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:46.971341 10604 server_base.cc:1034] running on GCE node
I20260924 23:16:46.971580 10604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:46.971849 10604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:46.972985 10604 hybrid_clock.cc:648] HybridClock initialized: now 1790291806972963 us; error 37 us; skew 500 ppm
I20260924 23:16:46.974285 10604 webserver.cc:460] Webserver started at http://127.1.4.195:35211/ using document root <none> and password file <none>
I20260924 23:16:46.974553 10604 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:46.974613 10604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:46.974745 10604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:46.976140 10604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "ce854f775d5f415aadaab66baac11f3f"
format_stamp: "Formatted at 2026-09-24 23:16:46 on dist-test-slave-s1hw"
server_key: "2677d03a913b0d7f2cb668b11db88360"
server_key_iv: "fb4b849db50bcab37e38664e5ffb5f56"
server_key_version: "encryptionkey@0"
I20260924 23:16:46.976524 10604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "ce854f775d5f415aadaab66baac11f3f"
format_stamp: "Formatted at 2026-09-24 23:16:46 on dist-test-slave-s1hw"
server_key: "2677d03a913b0d7f2cb668b11db88360"
server_key_iv: "fb4b849db50bcab37e38664e5ffb5f56"
server_key_version: "encryptionkey@0"
I20260924 23:16:46.978130 10604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 23:16:46.979192 10618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:46.979409 10604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:46.979512 10604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "ce854f775d5f415aadaab66baac11f3f"
format_stamp: "Formatted at 2026-09-24 23:16:46 on dist-test-slave-s1hw"
server_key: "2677d03a913b0d7f2cb668b11db88360"
server_key_iv: "fb4b849db50bcab37e38664e5ffb5f56"
server_key_version: "encryptionkey@0"
I20260924 23:16:46.979600 10604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:46.996716 10604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:46.997123 10604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:46.997264 10604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:46.997488 10604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:46.997797 10604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:46.997829 10604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:46.997852 10604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:46.997874 10604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:47.003234 10604 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:44715
I20260924 23:16:47.003283 10731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:44715 every 8 connection(s)
I20260924 23:16:47.003620 10604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:16:47.006902 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 10604
I20260924 23:16:47.006999 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:16:47.007186 1043 external_mini_cluster.cc:1428] Setting key 0c5dfa10bb112755069c429b3792a94a
I20260924 23:16:47.008838 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:47.004275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:59423 (local address 127.1.4.254:45587)
0924 23:16:47.004319 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:47.004321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:47.004557 (+ 236us) server_negotiation.cc:407] Connection header received
0924 23:16:47.005109 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:47.005111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:47.005122 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:47.005176 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:47.005832 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:47.006320 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:47.007557 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:47.007812 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:47.007868 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:47.008590 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:47.008592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:47.008592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:47.008596 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:47.008612 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:47.008616 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:47.008660 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:47.008734 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 23:16:47.008759 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:16:47.009085 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:47.004260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:45587 (local address 127.1.4.195:59423)
0924 23:16:47.004431 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:47.004470 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:47.005011 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:47.005243 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:47.005246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:47.005254 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:47.005744 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:47.005749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:47.006401 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:47.006402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:47.007472 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:47.007474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:47.007873 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:47.007874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:47.008015 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:47.008466 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:16:47.008467 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:47.008496 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:16:47.008519 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:47.008669 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:47.008673 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:47.008676 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:47.008734 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 23:16:47.008838 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260924 23:16:47.009583 10732 heartbeater.cc:347] Connected to a master server at 127.1.4.254:45587
I20260924 23:16:47.009661 10732 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:47.009828 10732 heartbeater.cc:511] Master 127.1.4.254:45587 requested a full tablet report, sending...
I20260924 23:16:47.010185 10281 ts_manager.cc:194] Registered new tserver with Master: ce854f775d5f415aadaab66baac11f3f (127.1.4.195:44715)
I20260924 23:16:47.010537 10281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.4.195:59423
I20260924 23:16:47.019039 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 23:16:47.020805 10321 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:47.020338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53824 (local address 127.1.4.254:45587)
0924 23:16:47.020392 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:47.020393 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:47.020507 (+ 114us) server_negotiation.cc:407] Connection header received
0924 23:16:47.020550 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:47.020551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:47.020559 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:47.020605 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:47.020731 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53824: BlockingRecv error: recv got EOF from 127.0.0.1:53824 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:16:47.020854 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 10342
I20260924 23:16:47.025065 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 10473
I20260924 23:16:47.028877 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 10604
I20260924 23:16:47.032534 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 10251
2026-09-24T23:16:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (602 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T23:16:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:47Z Disabled control of system clock
I20260924 23:16:47.057497 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:46449
--webserver_interface=127.1.4.254
--webserver_port=0
--builtin_ntp_servers=127.1.4.212:38205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:46449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 23:16:47.137271 10743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:47.137504 10743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:47.137534 10743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:47.138708 10743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:47.138751 10743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:47.138767 10743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:47.138783 10743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:47.138798 10743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:47.140184 10743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:38205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:46449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:46449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:47.140439 10743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:47.140708 10743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:47.145407 10751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:47.145424 10749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:47.145407 10748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:47.145613 10743 server_base.cc:1034] running on GCE node
I20260924 23:16:47.145864 10743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:47.146152 10743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:47.147301 10743 hybrid_clock.cc:648] HybridClock initialized: now 1790291807147285 us; error 36 us; skew 500 ppm
I20260924 23:16:47.148561 10743 webserver.cc:460] Webserver started at http://127.1.4.254:40541/ using document root <none> and password file <none>
I20260924 23:16:47.148770 10743 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:47.148818 10743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:47.148988 10743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:47.150048 10743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "3cb3a01bef874f60ae8a10ba7c9e6ec9"
format_stamp: "Formatted at 2026-09-24 23:16:47 on dist-test-slave-s1hw"
server_key: "25b08ec808e0acee1553182e24895140"
server_key_iv: "ee63882be03e92de896762577eec2f4c"
server_key_version: "encryptionkey@0"
I20260924 23:16:47.150360 10743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "3cb3a01bef874f60ae8a10ba7c9e6ec9"
format_stamp: "Formatted at 2026-09-24 23:16:47 on dist-test-slave-s1hw"
server_key: "25b08ec808e0acee1553182e24895140"
server_key_iv: "ee63882be03e92de896762577eec2f4c"
server_key_version: "encryptionkey@0"
I20260924 23:16:47.151659 10743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:47.152501 10757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:47.152666 10743 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:47.152758 10743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "3cb3a01bef874f60ae8a10ba7c9e6ec9"
format_stamp: "Formatted at 2026-09-24 23:16:47 on dist-test-slave-s1hw"
server_key: "25b08ec808e0acee1553182e24895140"
server_key_iv: "ee63882be03e92de896762577eec2f4c"
server_key_version: "encryptionkey@0"
I20260924 23:16:47.152818 10743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:47.156288 10743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:47.156540 10743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:47.156661 10743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:47.160211 10743 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:46449
I20260924 23:16:47.160249 10809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:46449 every 8 connection(s)
I20260924 23:16:47.160561 10743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:47.161283 10810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:47.161803 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 10743
I20260924 23:16:47.161996 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:47.162171 1043 external_mini_cluster.cc:1428] Setting key 0f9aa4e222ca86c43f7932040ea37b6a
I20260924 23:16:47.164614 10810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9: Bootstrap starting.
I20260924 23:16:47.164664 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:47.162698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.1.4.254:46449)
0924 23:16:47.162918 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:47.162925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:47.162952 (+ 27us) server_negotiation.cc:407] Connection header received
0924 23:16:47.163430 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:47.163439 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:47.163494 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:47.163619 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:47.164232 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:47.164235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:47.164235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:47.164238 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:47.164277 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:47.164288 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:47.164340 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:47.164425 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 23:16:47.164488 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260924 23:16:47.165282 10810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:47.165565 10810 log.cc:824] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:47.166191 10810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9: No bootstrap required, opened a new log
I20260924 23:16:47.167573 10810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb3a01bef874f60ae8a10ba7c9e6ec9" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46449 } }
I20260924 23:16:47.167711 10810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:47.167732 10810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb3a01bef874f60ae8a10ba7c9e6ec9, State: Initialized, Role: FOLLOWER
I20260924 23:16:47.167827 10810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb3a01bef874f60ae8a10ba7c9e6ec9" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46449 } }
I20260924 23:16:47.167910 10810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:47.167937 10810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:47.167970 10810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:47.168776 10810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb3a01bef874f60ae8a10ba7c9e6ec9" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46449 } }
I20260924 23:16:47.168864 10810 leader_election.cc:302] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cb3a01bef874f60ae8a10ba7c9e6ec9; no voters:
I20260924 23:16:47.169001 10810 leader_election.cc:288] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:47.169060 10815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:47.169217 10815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb3a01bef874f60ae8a10ba7c9e6ec9, State: Running, Role: LEADER
I20260924 23:16:47.169242 10810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:47.169315 10815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb3a01bef874f60ae8a10ba7c9e6ec9" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46449 } }
I20260924 23:16:47.169687 10816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cb3a01bef874f60ae8a10ba7c9e6ec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb3a01bef874f60ae8a10ba7c9e6ec9" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46449 } } }
I20260924 23:16:47.169725 10817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cb3a01bef874f60ae8a10ba7c9e6ec9. Latest consensus state: current_term: 1 leader_uuid: "3cb3a01bef874f60ae8a10ba7c9e6ec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb3a01bef874f60ae8a10ba7c9e6ec9" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46449 } } }
I20260924 23:16:47.169853 10817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:47.169807 10816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9 [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:47.170248 10824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:47.170686 10824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:47.172133 10824 catalog_manager.cc:1347] Generated new cluster ID: 87b274068a944a57bb1b3560df5728a0
I20260924 23:16:47.172178 10824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:16:47.188784 10824 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:47.189118 10824 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:47.196156 10824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cb3a01bef874f60ae8a10ba7c9e6ec9: Generated new TSK 0
I20260924 23:16:47.196362 10824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:16:47.198293 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:46449
--builtin_ntp_servers=127.1.4.212:38205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 23:16:47.281805 10834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:47.282034 10834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:47.282064 10834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:47.283277 10834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:47.283314 10834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:47.283349 10834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:16:47.284695 10834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:38205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:46449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:47.284925 10834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:47.285207 10834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:47.291008 10840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:47.291115 10834 server_base.cc:1034] running on GCE node
W20260924 23:16:47.291044 10842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:47.290987 10839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:47.291592 10834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:47.291831 10834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:47.293000 10834 hybrid_clock.cc:648] HybridClock initialized: now 1790291807292980 us; error 41 us; skew 500 ppm
I20260924 23:16:47.294312 10834 webserver.cc:460] Webserver started at http://127.1.4.193:41603/ using document root <none> and password file <none>
I20260924 23:16:47.294551 10834 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:47.294600 10834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:47.294730 10834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:47.295885 10834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "7982eebe98d34ad5afe055c2a34b9851"
format_stamp: "Formatted at 2026-09-24 23:16:47 on dist-test-slave-s1hw"
server_key: "63da749924b7cbcb23852be23ebd58ed"
server_key_iv: "2c685aa88aa3b4e69f1243030834cc2d"
server_key_version: "encryptionkey@0"
I20260924 23:16:47.296219 10834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "7982eebe98d34ad5afe055c2a34b9851"
format_stamp: "Formatted at 2026-09-24 23:16:47 on dist-test-slave-s1hw"
server_key: "63da749924b7cbcb23852be23ebd58ed"
server_key_iv: "2c685aa88aa3b4e69f1243030834cc2d"
server_key_version: "encryptionkey@0"
I20260924 23:16:47.297528 10834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:16:47.298439 10848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:47.298610 10834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:47.298696 10834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "7982eebe98d34ad5afe055c2a34b9851"
format_stamp: "Formatted at 2026-09-24 23:16:47 on dist-test-slave-s1hw"
server_key: "63da749924b7cbcb23852be23ebd58ed"
server_key_iv: "2c685aa88aa3b4e69f1243030834cc2d"
server_key_version: "encryptionkey@0"
I20260924 23:16:47.298779 10834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:47.302227 10834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:47.302474 10834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:47.302588 10834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:47.302780 10834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:47.303105 10834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:47.303138 10834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:47.303165 10834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:47.303187 10834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:47.308615 10834 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:41101
I20260924 23:16:47.308653 10961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:41101 every 8 connection(s)
I20260924 23:16:47.308974 10834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:16:47.311904 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:47.309651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:48199 (local address 127.1.4.254:46449)
0924 23:16:47.309696 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:47.309698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:47.309908 (+ 210us) server_negotiation.cc:407] Connection header received
0924 23:16:47.310413 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:47.310414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:47.310422 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:47.310474 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:47.311670 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:47.311672 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:47.311672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:47.311676 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:47.311692 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:47.311696 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:47.311745 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:47.311819 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 23:16:47.311833 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:16:47.312091 10964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:47.309620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:46449 (local address 127.1.4.193:48199)
0924 23:16:47.309784 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:47.309818 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:47.310310 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:47.310580 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:47.310583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:47.310591 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:47.311525 (+ 934us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:16:47.311526 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:47.311556 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:16:47.311586 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:47.311749 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:47.311753 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:47.311757 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:47.311827 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 23:16:47.311889 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
I20260924 23:16:47.312611 10962 heartbeater.cc:347] Connected to a master server at 127.1.4.254:46449
I20260924 23:16:47.312711 10962 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:47.312851 10962 heartbeater.cc:511] Master 127.1.4.254:46449 requested a full tablet report, sending...
I20260924 23:16:47.313198 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 10834
I20260924 23:16:47.313277 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:16:47.313390 10774 ts_manager.cc:194] Registered new tserver with Master: 7982eebe98d34ad5afe055c2a34b9851 (127.1.4.193:41101)
I20260924 23:16:47.313467 1043 external_mini_cluster.cc:1428] Setting key 49f05eb30e9de1e109af01c8149772c7
I20260924 23:16:47.314334 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:46449
--builtin_ntp_servers=127.1.4.212:38205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 23:16:47.393775 10965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:47.393999 10965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:47.394022 10965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:47.395144 10965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:47.395181 10965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:47.395205 10965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:16:47.396570 10965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:38205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:46449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:47.396782 10965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:47.397032 10965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:47.402027 10970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:47.402087 10971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:47.402187 10965 server_base.cc:1034] running on GCE node
W20260924 23:16:47.401912 10973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:47.402535 10965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:47.402752 10965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:47.403887 10965 hybrid_clock.cc:648] HybridClock initialized: now 1790291807403873 us; error 28 us; skew 500 ppm
I20260924 23:16:47.405133 10965 webserver.cc:460] Webserver started at http://127.1.4.194:45851/ using document root <none> and password file <none>
I20260924 23:16:47.405349 10965 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:47.405391 10965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:47.405532 10965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:47.406589 10965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "1fa46199a1654c17b119d0ae55a75d0e"
format_stamp: "Formatted at 2026-09-24 23:16:47 on dist-test-slave-s1hw"
server_key: "9dfb1b95b1b8f93375ba8456b72e5cdf"
server_key_iv: "e6d3ed2bf78bb8f292265efd427002b4"
server_key_version: "encryptionkey@0"
I20260924 23:16:47.406899 10965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "1fa46199a1654c17b119d0ae55a75d0e"
format_stamp: "Formatted at 2026-09-24 23:16:47 on dist-test-slave-s1hw"
server_key: "9dfb1b95b1b8f93375ba8456b72e5cdf"
server_key_iv: "e6d3ed2bf78bb8f292265efd427002b4"
server_key_version: "encryptionkey@0"
I20260924 23:16:47.408239 10965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:16:47.408964 10979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:47.409123 10965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:16:47.409200 10965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "1fa46199a1654c17b119d0ae55a75d0e"
format_stamp: "Formatted at 2026-09-24 23:16:47 on dist-test-slave-s1hw"
server_key: "9dfb1b95b1b8f93375ba8456b72e5cdf"
server_key_iv: "e6d3ed2bf78bb8f292265efd427002b4"
server_key_version: "encryptionkey@0"
I20260924 23:16:47.409260 10965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:47.412545 10965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:47.412786 10965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:47.412904 10965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:47.413089 10965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:47.413378 10965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:47.413416 10965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:47.413452 10965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:47.413480 10965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:47.418793 10965 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:38581
I20260924 23:16:47.418838 11092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:38581 every 8 connection(s)
I20260924 23:16:47.419152 10965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:16:47.421957 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:47.419875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:45661 (local address 127.1.4.254:46449)
0924 23:16:47.419962 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:47.419964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:47.420164 (+ 200us) server_negotiation.cc:407] Connection header received
0924 23:16:47.420662 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:47.420663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:47.420668 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:47.420730 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:47.421770 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:47.421771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:47.421771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:47.421775 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:47.421789 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:47.421792 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:47.421838 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:47.421888 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 23:16:47.421899 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 23:16:47.422122 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:47.419851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:46449 (local address 127.1.4.194:45661)
0924 23:16:47.420036 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:47.420077 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:47.420559 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:47.420786 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:47.420789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:47.420797 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:47.421630 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:16:47.421630 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:47.421659 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:16:47.421688 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:47.421836 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:47.421838 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:47.421841 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:47.421891 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 23:16:47.421939 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260924 23:16:47.422541 11093 heartbeater.cc:347] Connected to a master server at 127.1.4.254:46449
I20260924 23:16:47.422626 11093 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:47.422767 11093 heartbeater.cc:511] Master 127.1.4.254:46449 requested a full tablet report, sending...
I20260924 23:16:47.423105 10774 ts_manager.cc:194] Registered new tserver with Master: 1fa46199a1654c17b119d0ae55a75d0e (127.1.4.194:38581)
I20260924 23:16:47.428819 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 10965
I20260924 23:16:47.428920 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:16:47.429096 1043 external_mini_cluster.cc:1428] Setting key b7d131bf9b92d3195f90ae7c9d0476f5
I20260924 23:16:47.430138 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:46449
--builtin_ntp_servers=127.1.4.212:38205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 23:16:47.510195 11096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:47.510404 11096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:47.510425 11096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:47.511654 11096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:47.511693 11096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:47.511718 11096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:16:47.513034 11096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:38205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:46449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:47.513221 11096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:47.513454 11096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:47.518227 11101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:47.518209 11104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:47.518215 11102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:47.518282 11096 server_base.cc:1034] running on GCE node
I20260924 23:16:47.518625 11096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:47.518838 11096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:47.519994 11096 hybrid_clock.cc:648] HybridClock initialized: now 1790291807519980 us; error 30 us; skew 500 ppm
I20260924 23:16:47.521174 11096 webserver.cc:460] Webserver started at http://127.1.4.195:41943/ using document root <none> and password file <none>
I20260924 23:16:47.521375 11096 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:47.521418 11096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:47.521533 11096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:47.522531 11096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "924816b0078548ccaa899a767e7fca58"
format_stamp: "Formatted at 2026-09-24 23:16:47 on dist-test-slave-s1hw"
server_key: "6786b3afc75cde3da8b59a52dcd60738"
server_key_iv: "13f2e8c8c2a71f39584c68f24d9c81f2"
server_key_version: "encryptionkey@0"
I20260924 23:16:47.522840 11096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "924816b0078548ccaa899a767e7fca58"
format_stamp: "Formatted at 2026-09-24 23:16:47 on dist-test-slave-s1hw"
server_key: "6786b3afc75cde3da8b59a52dcd60738"
server_key_iv: "13f2e8c8c2a71f39584c68f24d9c81f2"
server_key_version: "encryptionkey@0"
I20260924 23:16:47.524179 11096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:16:47.524830 11110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:47.524982 11096 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:16:47.525055 11096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "924816b0078548ccaa899a767e7fca58"
format_stamp: "Formatted at 2026-09-24 23:16:47 on dist-test-slave-s1hw"
server_key: "6786b3afc75cde3da8b59a52dcd60738"
server_key_iv: "13f2e8c8c2a71f39584c68f24d9c81f2"
server_key_version: "encryptionkey@0"
I20260924 23:16:47.525120 11096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:47.528434 11096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:47.528663 11096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:47.528775 11096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:47.528964 11096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:47.529232 11096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:47.529263 11096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:47.529281 11096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:47.529313 11096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:47.534667 11096 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:34627
I20260924 23:16:47.534727 11223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:34627 every 8 connection(s)
I20260924 23:16:47.535017 11096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:16:47.538055 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:47.535756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:46451 (local address 127.1.4.254:46449)
0924 23:16:47.535818 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:47.535820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:47.536076 (+ 256us) server_negotiation.cc:407] Connection header received
0924 23:16:47.536642 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:47.536644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:47.536651 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:47.536711 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:47.537851 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:47.537852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:47.537852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:16:47.537856 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:47.537870 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:16:47.537873 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:47.537917 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:47.537984 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 23:16:47.537998 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 23:16:47.538231 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:47.535717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:46449 (local address 127.1.4.195:46451)
0924 23:16:47.535958 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:47.535994 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:47.536490 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:47.536781 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:47.536784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:47.536793 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:47.537708 (+ 915us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:16:47.537709 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:47.537739 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:16:47.537768 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:47.537926 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:47.537928 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:47.537931 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:47.537981 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 23:16:47.538031 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20260924 23:16:47.538625 11224 heartbeater.cc:347] Connected to a master server at 127.1.4.254:46449
I20260924 23:16:47.538707 11224 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:47.538843 11224 heartbeater.cc:511] Master 127.1.4.254:46449 requested a full tablet report, sending...
I20260924 23:16:47.539232 10774 ts_manager.cc:194] Registered new tserver with Master: 924816b0078548ccaa899a767e7fca58 (127.1.4.195:34627)
I20260924 23:16:47.544945 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 11096
I20260924 23:16:47.545043 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:16:47.545213 1043 external_mini_cluster.cc:1428] Setting key 4dac9985ed76f417829fb078f6fc2d12
I20260924 23:16:47.546059 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 23:16:47.547768 10813 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:47.547315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35582 (local address 127.1.4.254:46449)
0924 23:16:47.547406 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:47.547408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:47.547465 (+ 57us) server_negotiation.cc:407] Connection header received
0924 23:16:47.547527 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:47.547528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:47.547533 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:47.547580 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:47.547699 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35582: BlockingRecv error: recv got EOF from 127.0.0.1:35582 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 23:16:47.547816 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 10834
I20260924 23:16:47.552246 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 10965
I20260924 23:16:47.556187 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 11096
I20260924 23:16:47.560249 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 10743
2026-09-24T23:16:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (527 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:16:47 dist-test-slave-s1hw krb5kdc[11234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:16:47 dist-test-slave-s1hw krb5kdc[11234](info): set up 2 sockets
Sep 24 23:16:47 dist-test-slave-s1hw krb5kdc[11234](info): commencing operation
krb5kdc: starting...
W20260924 23:16:48.519933 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:16:48 dist-test-slave-s1hw krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:16:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:16:48.634685 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:33945
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:33945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:48.713176 11247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:48.713373 11247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:48.713393 11247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:48.714475 11247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:16:48.714509 11247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:48.714522 11247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:48.714533 11247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:16:48.714550 11247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:16:48.715994 11247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:33945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:33945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:48.716180 11247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:48.716351 11247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:48.720882 11253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:48.720934 11252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:48.720894 11255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:48.721100 11247 server_base.cc:1034] running on GCE node
I20260924 23:16:48.721397 11247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:48.721714 11247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:48.722851 11247 hybrid_clock.cc:648] HybridClock initialized: now 1790291808722832 us; error 32 us; skew 500 ppm
Sep 24 23:16:48 dist-test-slave-s1hw krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291808, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:48.725251 11247 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:16:48.725701 11247 webserver.cc:460] Webserver started at http://127.1.4.254:44511/ using document root <none> and password file <none>
I20260924 23:16:48.725917 11247 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:48.725979 11247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:48.726094 11247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:48.727097 11247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "fba529caab2442d8af83ac54d86fc28d"
format_stamp: "Formatted at 2026-09-24 23:16:48 on dist-test-slave-s1hw"
server_key: "b38a2929e9f4a1e2f2a75657d00d52c7"
server_key_iv: "2bb09764ed08ffc9dde13f0d58977ca7"
server_key_version: "encryptionkey@0"
I20260924 23:16:48.727447 11247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "fba529caab2442d8af83ac54d86fc28d"
format_stamp: "Formatted at 2026-09-24 23:16:48 on dist-test-slave-s1hw"
server_key: "b38a2929e9f4a1e2f2a75657d00d52c7"
server_key_iv: "2bb09764ed08ffc9dde13f0d58977ca7"
server_key_version: "encryptionkey@0"
I20260924 23:16:48.728657 11247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:16:48.729467 11262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:48.729627 11247 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:16:48.729712 11247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "fba529caab2442d8af83ac54d86fc28d"
format_stamp: "Formatted at 2026-09-24 23:16:48 on dist-test-slave-s1hw"
server_key: "b38a2929e9f4a1e2f2a75657d00d52c7"
server_key_iv: "2bb09764ed08ffc9dde13f0d58977ca7"
server_key_version: "encryptionkey@0"
I20260924 23:16:48.729773 11247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:48.744055 11247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:48.745966 11247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:48.746105 11247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:48.749994 11247 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:33945
I20260924 23:16:48.750044 11314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:33945 every 8 connection(s)
I20260924 23:16:48.750385 11247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:16:48.751335 11315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:48.753590 11315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d: Bootstrap starting.
I20260924 23:16:48.754142 11315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:48.754386 11315 log.cc:824] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:48.755071 11315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d: No bootstrap required, opened a new log
I20260924 23:16:48.756641 11315 raft_consensus.cc:348] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba529caab2442d8af83ac54d86fc28d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33945 } }
I20260924 23:16:48.756778 11315 raft_consensus.cc:374] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:48.756798 11315 raft_consensus.cc:729] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba529caab2442d8af83ac54d86fc28d, State: Initialized, Role: FOLLOWER
I20260924 23:16:48.756938 11315 consensus_queue.cc:260] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba529caab2442d8af83ac54d86fc28d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33945 } }
I20260924 23:16:48.757009 11315 raft_consensus.cc:388] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:16:48.757031 11315 raft_consensus.cc:482] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:16:48.757074 11315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:48.757825 11315 raft_consensus.cc:504] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba529caab2442d8af83ac54d86fc28d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33945 } }
I20260924 23:16:48.757910 11315 leader_election.cc:302] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fba529caab2442d8af83ac54d86fc28d; no voters:
I20260924 23:16:48.758070 11315 leader_election.cc:288] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:16:48.758107 11318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:48.758257 11318 raft_consensus.cc:686] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [term 1 LEADER]: Becoming Leader. State: Replica: fba529caab2442d8af83ac54d86fc28d, State: Running, Role: LEADER
I20260924 23:16:48.758311 11315 sys_catalog.cc:564] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:16:48.758347 11318 consensus_queue.cc:237] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba529caab2442d8af83ac54d86fc28d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33945 } }
I20260924 23:16:48.758679 11320 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fba529caab2442d8af83ac54d86fc28d. Latest consensus state: current_term: 1 leader_uuid: "fba529caab2442d8af83ac54d86fc28d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba529caab2442d8af83ac54d86fc28d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33945 } } }
I20260924 23:16:48.758760 11320 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:48.758669 11319 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fba529caab2442d8af83ac54d86fc28d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba529caab2442d8af83ac54d86fc28d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33945 } } }
I20260924 23:16:48.758909 11319 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d [sys.catalog]: This master's current role is: LEADER
I20260924 23:16:48.758980 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 11247
I20260924 23:16:48.759075 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:16:48.759246 1043 external_mini_cluster.cc:1428] Setting key 99a00303c3de8bc8d88d7c7dfa2778ed
I20260924 23:16:48.759262 11325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:16:48.760368 11325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:16:48.761765 11325 catalog_manager.cc:1347] Generated new cluster ID: c2cdd9b5ea5747f9afc5c356389ff06e
I20260924 23:16:48.761814 11325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:16:48 dist-test-slave-s1hw krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:48.770725 11335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:48.759921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.1.4.254:33945)
0924 23:16:48.760068 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:48.760078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:48.760117 (+ 39us) server_negotiation.cc:407] Connection header received
0924 23:16:48.760235 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:48.760243 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:48.760301 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:48.760455 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:48.761262 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:48.762632 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:48.763691 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:48.764374 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:48.764495 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:48.767106 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:48.767110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:48.767111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:48.767123 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:48.769064 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:48.769482 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:48.769485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:48.769486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:48.769590 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:48.769766 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:48.769767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:48.769767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:48.770085 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:48.770228 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:48.770340 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 23:16:48.770473 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":66,"threads_started":1}
I20260924 23:16:48.772539 11325 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:16:48.773017 11325 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:16:48.780408 11325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d: Generated new TSK 0
I20260924 23:16:48.780606 11325 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:16:48.810612 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33945
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:48.899559 11341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:48.899763 11341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:48.899787 11341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:48.900933 11341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:48.900974 11341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:48.900998 11341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:16:48.902411 11341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:48.902617 11341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:48.902791 11341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:48.907804 11346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:48.907833 11347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:48.907815 11349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:48.908030 11341 server_base.cc:1034] running on GCE node
I20260924 23:16:48.908206 11341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:48.908480 11341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:48.909627 11341 hybrid_clock.cc:648] HybridClock initialized: now 1790291808909612 us; error 29 us; skew 500 ppm
Sep 24 23:16:48 dist-test-slave-s1hw krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291808, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:48.917029 11341 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:16:48.917553 11341 webserver.cc:460] Webserver started at http://127.1.4.193:36771/ using document root <none> and password file <none>
I20260924 23:16:48.917755 11341 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:48.917802 11341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:48.917927 11341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:48.918957 11341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "ebe675f1a75841f5949a6f947e8eb7f4"
format_stamp: "Formatted at 2026-09-24 23:16:48 on dist-test-slave-s1hw"
server_key: "afaab3b0370300caa017c82e7e54db43"
server_key_iv: "1d72fce81918a4ab6843df73ad4fde7b"
server_key_version: "encryptionkey@0"
I20260924 23:16:48.919281 11341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "ebe675f1a75841f5949a6f947e8eb7f4"
format_stamp: "Formatted at 2026-09-24 23:16:48 on dist-test-slave-s1hw"
server_key: "afaab3b0370300caa017c82e7e54db43"
server_key_iv: "1d72fce81918a4ab6843df73ad4fde7b"
server_key_version: "encryptionkey@0"
I20260924 23:16:48.920563 11341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:16:48.921231 11356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:48.921386 11341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:16:48.921495 11341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "ebe675f1a75841f5949a6f947e8eb7f4"
format_stamp: "Formatted at 2026-09-24 23:16:48 on dist-test-slave-s1hw"
server_key: "afaab3b0370300caa017c82e7e54db43"
server_key_iv: "1d72fce81918a4ab6843df73ad4fde7b"
server_key_version: "encryptionkey@0"
I20260924 23:16:48.921556 11341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:48.941978 11341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:48.944154 11341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:48.944288 11341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:48.944512 11341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:48.944796 11341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:48.944828 11341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:48.944856 11341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:48.944880 11341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:48.950297 11341 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:42605
I20260924 23:16:48.950345 11469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:42605 every 8 connection(s)
I20260924 23:16:48.950666 11341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:16:48.956100 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 11341
I20260924 23:16:48.956198 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:16:48.956372 1043 external_mini_cluster.cc:1428] Setting key 8580999a1d292ae08a3de204547ef169
Sep 24 23:16:48 dist-test-slave-s1hw krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291808, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:48.960067 11335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:48.951417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:51651 (local address 127.1.4.254:33945)
0924 23:16:48.951489 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:48.951491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:48.951638 (+ 147us) server_negotiation.cc:407] Connection header received
0924 23:16:48.952135 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:48.952136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:48.952146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:48.952196 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:48.953097 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:48.953597 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:48.954623 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:48.954821 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:48.954873 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:48.957161 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:48.957162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:48.957163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:48.957170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:48.958620 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:48.959048 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:48.959049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:48.959049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:48.959112 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:48.959399 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:48.959400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:48.959400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:48.959516 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:48.959568 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:48.959920 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 23:16:48.960001 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 23:16:48.960294 11472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:48.951345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33945 (local address 127.1.4.193:51651)
0924 23:16:48.951520 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:48.951558 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:48.952048 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:48.952251 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:48.952254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:48.952809 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:48.953003 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:48.953008 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:48.953686 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:48.953687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:48.954545 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:48.954548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:48.954912 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:48.954913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:48.955030 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:48.955574 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:48.955583 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:48.957072 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:48.958701 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:48.958708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:48.958710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:48.958933 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:48.959175 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:48.959176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:48.959176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:48.959309 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:48.959573 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:48.959579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:48.959746 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 23:16:48.959850 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 23:16:48.960062 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260924 23:16:48.960803 11470 heartbeater.cc:347] Connected to a master server at 127.1.4.254:33945
I20260924 23:16:48.960893 11470 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:48.961076 11470 heartbeater.cc:511] Master 127.1.4.254:33945 requested a full tablet report, sending...
I20260924 23:16:48.961608 11279 ts_manager.cc:194] Registered new tserver with Master: ebe675f1a75841f5949a6f947e8eb7f4 (127.1.4.193:42605)
I20260924 23:16:48.962512 11279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:51651
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:16:48.982570 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33945
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:49.077585 11475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:49.077806 11475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:49.077832 11475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:49.078985 11475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:49.079027 11475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:49.079053 11475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:16:49.080592 11475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:49.080821 11475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:49.081054 11475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:16:49.086329 11480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:49.086359 11481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:49.086287 11483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:49.086333 11475 server_base.cc:1034] running on GCE node
I20260924 23:16:49.086720 11475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:49.086967 11475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:49.088105 11475 hybrid_clock.cc:648] HybridClock initialized: now 1790291809088085 us; error 36 us; skew 500 ppm
Sep 24 23:16:49 dist-test-slave-s1hw krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291809, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:49.090662 11475 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:16:49.091082 11475 webserver.cc:460] Webserver started at http://127.1.4.194:42709/ using document root <none> and password file <none>
I20260924 23:16:49.091288 11475 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:49.091331 11475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:49.091495 11475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:49.092527 11475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "2a1d04990b2f46f4a7e136007857d0f7"
format_stamp: "Formatted at 2026-09-24 23:16:49 on dist-test-slave-s1hw"
server_key: "b63fe8729f3867668da41620ef517dde"
server_key_iv: "86104ddb3d45636517f4a592913c10c3"
server_key_version: "encryptionkey@0"
I20260924 23:16:49.092846 11475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "2a1d04990b2f46f4a7e136007857d0f7"
format_stamp: "Formatted at 2026-09-24 23:16:49 on dist-test-slave-s1hw"
server_key: "b63fe8729f3867668da41620ef517dde"
server_key_iv: "86104ddb3d45636517f4a592913c10c3"
server_key_version: "encryptionkey@0"
I20260924 23:16:49.094064 11475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:16:49.094857 11490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:49.095005 11475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:16:49.095086 11475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "2a1d04990b2f46f4a7e136007857d0f7"
format_stamp: "Formatted at 2026-09-24 23:16:49 on dist-test-slave-s1hw"
server_key: "b63fe8729f3867668da41620ef517dde"
server_key_iv: "86104ddb3d45636517f4a592913c10c3"
server_key_version: "encryptionkey@0"
I20260924 23:16:49.095144 11475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:49.108222 11475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:49.110492 11475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:49.110653 11475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:49.110875 11475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:49.111191 11475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:49.111224 11475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:49.111253 11475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:49.111269 11475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:49.116700 11475 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:45873
I20260924 23:16:49.116752 11603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:45873 every 8 connection(s)
I20260924 23:16:49.117110 11475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:16:49 dist-test-slave-s1hw krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291809, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:49.126120 11335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:49.117793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:44133 (local address 127.1.4.254:33945)
0924 23:16:49.117837 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:49.117839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:49.118041 (+ 202us) server_negotiation.cc:407] Connection header received
0924 23:16:49.118596 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:49.118597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:49.118610 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:49.118661 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:49.119617 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.120111 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.121104 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.121299 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.121356 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:49.123415 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:49.123417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:49.123417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:49.123425 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:49.124728 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:49.125111 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:49.125112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:49.125113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:49.125183 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:49.125484 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:49.125485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:49.125485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:49.125627 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:49.125699 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:49.125980 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 23:16:49.126055 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 23:16:49.126217 11606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:49.117765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33945 (local address 127.1.4.194:44133)
0924 23:16:49.117895 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:49.117942 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:49.118466 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:49.118733 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:49.118737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:49.119286 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:49.119509 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:49.119513 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.120195 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.120196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:49.121013 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:49.121016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.121367 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.121368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:49.121495 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:49.121932 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:49.121940 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:49.123293 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:49.124801 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:49.124807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:49.124810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:49.125041 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:49.125259 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:49.125260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:49.125260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:49.125377 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:49.125688 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:49.125690 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:49.125811 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 23:16:49.125891 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 23:16:49.126033 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20260924 23:16:49.126679 11604 heartbeater.cc:347] Connected to a master server at 127.1.4.254:33945
I20260924 23:16:49.126765 11604 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:49.126920 11604 heartbeater.cc:511] Master 127.1.4.254:33945 requested a full tablet report, sending...
I20260924 23:16:49.127228 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 11475
I20260924 23:16:49.127321 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:16:49.127439 11279 ts_manager.cc:194] Registered new tserver with Master: 2a1d04990b2f46f4a7e136007857d0f7 (127.1.4.194:45873)
I20260924 23:16:49.127539 1043 external_mini_cluster.cc:1428] Setting key 9c15c258b5124d4ca78e3c0ac57b57f4
I20260924 23:16:49.127935 11279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:44133
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:16:49.152174 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33945
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:16:49.231294 11609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:16:49.231559 11609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:16:49.231592 11609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:16:49.232800 11609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:16:49.232839 11609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:16:49.232874 11609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:16:49.234354 11609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:16:49.234591 11609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:16:49.234794 11609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:16:49.239647 11609 server_base.cc:1034] running on GCE node
W20260924 23:16:49.239727 11615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:49.239741 11614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:16:49.239890 11617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:16:49.240099 11609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:16:49.240300 11609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:16:49.241446 11609 hybrid_clock.cc:648] HybridClock initialized: now 1790291809241433 us; error 24 us; skew 500 ppm
Sep 24 23:16:49 dist-test-slave-s1hw krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291809, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:16:49.243831 11609 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:16:49.244269 11609 webserver.cc:460] Webserver started at http://127.1.4.195:35149/ using document root <none> and password file <none>
I20260924 23:16:49.244463 11609 fs_manager.cc:359] Metadata directory not provided
I20260924 23:16:49.244505 11609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:16:49.244619 11609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:16:49.245643 11609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "49cfad4fe59d43f5822acd19245fb41a"
format_stamp: "Formatted at 2026-09-24 23:16:49 on dist-test-slave-s1hw"
server_key: "6d70d317de02f4c1e775d469abcb455e"
server_key_iv: "cc236794af7e2a5310e553088e2f9a62"
server_key_version: "encryptionkey@0"
I20260924 23:16:49.245958 11609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "49cfad4fe59d43f5822acd19245fb41a"
format_stamp: "Formatted at 2026-09-24 23:16:49 on dist-test-slave-s1hw"
server_key: "6d70d317de02f4c1e775d469abcb455e"
server_key_iv: "cc236794af7e2a5310e553088e2f9a62"
server_key_version: "encryptionkey@0"
I20260924 23:16:49.247164 11609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:16:49.247853 11624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:49.248032 11609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:49.248104 11609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "49cfad4fe59d43f5822acd19245fb41a"
format_stamp: "Formatted at 2026-09-24 23:16:49 on dist-test-slave-s1hw"
server_key: "6d70d317de02f4c1e775d469abcb455e"
server_key_iv: "cc236794af7e2a5310e553088e2f9a62"
server_key_version: "encryptionkey@0"
I20260924 23:16:49.248168 11609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:16:49.255934 11609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:16:49.257802 11609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:16:49.257916 11609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:16:49.258122 11609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:16:49.258424 11609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:16:49.258457 11609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:49.258477 11609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:16:49.258496 11609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:16:49.264092 11609 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:46013
I20260924 23:16:49.264140 11737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:46013 every 8 connection(s)
I20260924 23:16:49.264438 11609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:16:49.266568 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 11609
I20260924 23:16:49.266664 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:16:49.266830 1043 external_mini_cluster.cc:1428] Setting key 475af93df428deebcd5ffe4381e16f74
Sep 24 23:16:49 dist-test-slave-s1hw krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291809, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:16:49.273556 11335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:49.265095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:50915 (local address 127.1.4.254:33945)
0924 23:16:49.265146 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:49.265148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:49.265350 (+ 202us) server_negotiation.cc:407] Connection header received
0924 23:16:49.265836 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:49.265837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:49.265850 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:49.265901 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:49.266859 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.267545 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.268516 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.268701 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.268752 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:49.270978 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:49.270979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:49.270980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:49.270987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:49.272254 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:49.272636 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:49.272637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:49.272638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:49.272688 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:49.272922 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:49.272923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:49.272924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:49.273034 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:49.273090 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:49.273370 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 23:16:49.273452 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 23:16:49.273715 11740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:49.265070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33945 (local address 127.1.4.195:50915)
0924 23:16:49.265234 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:49.265269 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:49.265748 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:49.265979 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:49.265983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:49.266540 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:16:49.266768 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:49.266774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.267630 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.267631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:49.268447 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:49.268449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.268759 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.268760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:49.268853 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:49.269363 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:16:49.269371 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:16:49.270782 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:16:49.272337 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:49.272344 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:49.272346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:49.272568 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:49.272751 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:16:49.272751 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:16:49.272752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:16:49.272865 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:16:49.273088 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:16:49.273094 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:16:49.273235 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 23:16:49.273322 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 23:16:49.273481 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260924 23:16:49.274166 11738 heartbeater.cc:347] Connected to a master server at 127.1.4.254:33945
I20260924 23:16:49.274250 11738 heartbeater.cc:464] Registering TS with master...
I20260924 23:16:49.274441 11738 heartbeater.cc:511] Master 127.1.4.254:33945 requested a full tablet report, sending...
I20260924 23:16:49.274803 11278 ts_manager.cc:194] Registered new tserver with Master: 49cfad4fe59d43f5822acd19245fb41a (127.1.4.195:46013)
I20260924 23:16:49.275204 11278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:50915
I20260924 23:16:49.278373 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:16:49.285617 11335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:49.279633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.1.4.254:33945)
0924 23:16:49.279697 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:49.279699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:49.279772 (+ 73us) server_negotiation.cc:407] Connection header received
0924 23:16:49.279812 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:49.279813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:49.279821 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:49.279864 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:16:49.280436 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.280953 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.281773 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.281955 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.282041 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:49.283235 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:16:49.283236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:16:49.283236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:16:49.283243 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:16:49.284605 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:49.284935 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:49.284936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:49.284937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:49.285031 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:16:49.285189 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:16:49.285190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:16:49.285190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:16:49.285329 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:16:49.285401 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:49.285492 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 23:16:49.285551 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 23:16:49.287062 11278 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:16:49.288698 11278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:16:49.294678 11749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:49.290865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:42605 (local address 127.0.0.1:38716)
0924 23:16:49.291183 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:49.291192 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:49.291234 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:49.291444 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:49.291448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:49.291450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:49.291551 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:49.291553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.292506 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.292507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:49.293532 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:49.293535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.294385 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.294386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:49.294458 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:49.294481 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 23:16:49.294546 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 23:16:49.294598 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":132,"threads_started":1}
I20260924 23:16:49.294884 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:49.291019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.1.4.194:45873)
0924 23:16:49.291484 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:49.291488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:49.291496 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:49.291530 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:49.291531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:49.291545 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:49.291650 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:49.291761 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.292843 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.293911 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.294475 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.294543 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:49.294596 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:49.294663 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 23:16:49.294700 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":86,"threads_started":1}
I20260924 23:16:49.294770 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:49.290845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:45873 (local address 127.0.0.1:55092)
0924 23:16:49.291175 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:49.291186 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:49.291238 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:49.291654 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:49.291655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:49.291657 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:49.291706 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:49.291709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.292951 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.292952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:49.293835 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:49.293837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.294550 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.294551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:49.294602 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:49.294605 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:49.294653 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 23:16:49.294685 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":41,"thread_start_us":89,"threads_started":1}
I20260924 23:16:49.294997 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:49.291135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.1.4.193:42605)
0924 23:16:49.291284 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:49.291289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:49.291298 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:16:49.291335 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:49.291337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:49.291352 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:49.291435 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:49.291632 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.292417 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.293623 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.294318 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.294374 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:49.294435 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:49.294884 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 23:16:49.294925 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260924 23:16:49.295152 11750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:49.290845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:46013 (local address 127.0.0.1:35188)
0924 23:16:49.291009 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:49.291047 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:49.291110 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:49.291527 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:49.291528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:49.291530 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:49.291842 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:49.291845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.293043 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.293044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:49.294099 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:49.294101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.294937 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.294938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:49.295005 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:49.295009 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:16:49.295036 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 23:16:49.295078 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":44,"threads_started":1}
I20260924 23:16:49.295522 11752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:49.291006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.1.4.195:46013)
0924 23:16:49.291339 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:49.291343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:49.291353 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:16:49.291402 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:49.291404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:49.291420 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:49.291526 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:49.291910 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.292951 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.294175 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.294844 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.295301 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:49.295396 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:49.295430 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 23:16:49.295465 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":87,"threads_started":1}
I20260924 23:16:49.295691 11404 tablet_service.cc:1467] Processing CreateTablet for tablet 64a60f4b71464081993db3b6cc5b3eed (DEFAULT_TABLE table=test-table [id=deef3cc87c6b466db987fc0e2f801cd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:49.295689 11538 tablet_service.cc:1467] Processing CreateTablet for tablet 64a60f4b71464081993db3b6cc5b3eed (DEFAULT_TABLE table=test-table [id=deef3cc87c6b466db987fc0e2f801cd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:49.296025 11404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a60f4b71464081993db3b6cc5b3eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:49.296051 11538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a60f4b71464081993db3b6cc5b3eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:49.299325 11756 tablet_bootstrap.cc:492] T 64a60f4b71464081993db3b6cc5b3eed P ebe675f1a75841f5949a6f947e8eb7f4: Bootstrap starting.
I20260924 23:16:49.299749 11672 tablet_service.cc:1467] Processing CreateTablet for tablet 64a60f4b71464081993db3b6cc5b3eed (DEFAULT_TABLE table=test-table [id=deef3cc87c6b466db987fc0e2f801cd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:16:49.299954 11756 tablet_bootstrap.cc:654] T 64a60f4b71464081993db3b6cc5b3eed P ebe675f1a75841f5949a6f947e8eb7f4: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:49.300009 11672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a60f4b71464081993db3b6cc5b3eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:16:49.300395 11756 log.cc:824] T 64a60f4b71464081993db3b6cc5b3eed P ebe675f1a75841f5949a6f947e8eb7f4: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:49.300721 11755 tablet_bootstrap.cc:492] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7: Bootstrap starting.
I20260924 23:16:49.301224 11755 tablet_bootstrap.cc:654] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:49.301640 11755 log.cc:824] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:49.302140 11756 tablet_bootstrap.cc:492] T 64a60f4b71464081993db3b6cc5b3eed P ebe675f1a75841f5949a6f947e8eb7f4: No bootstrap required, opened a new log
I20260924 23:16:49.302210 11756 ts_tablet_manager.cc:1397] T 64a60f4b71464081993db3b6cc5b3eed P ebe675f1a75841f5949a6f947e8eb7f4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 23:16:49.302457 11759 tablet_bootstrap.cc:492] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a: Bootstrap starting.
I20260924 23:16:49.303049 11759 tablet_bootstrap.cc:654] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a: Neither blocks nor log segments found. Creating new log.
I20260924 23:16:49.303583 11759 log.cc:824] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:16:49.303684 11755 tablet_bootstrap.cc:492] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7: No bootstrap required, opened a new log
I20260924 23:16:49.303750 11755 ts_tablet_manager.cc:1397] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 23:16:49.304141 11756 raft_consensus.cc:348] T 64a60f4b71464081993db3b6cc5b3eed P ebe675f1a75841f5949a6f947e8eb7f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49cfad4fe59d43f5822acd19245fb41a" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46013 } } peers { permanent_uuid: "2a1d04990b2f46f4a7e136007857d0f7" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45873 } } peers { permanent_uuid: "ebe675f1a75841f5949a6f947e8eb7f4" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42605 } }
I20260924 23:16:49.304333 11759 tablet_bootstrap.cc:492] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a: No bootstrap required, opened a new log
I20260924 23:16:49.304391 11759 ts_tablet_manager.cc:1397] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 23:16:49.304430 11756 raft_consensus.cc:374] T 64a60f4b71464081993db3b6cc5b3eed P ebe675f1a75841f5949a6f947e8eb7f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:49.304484 11756 raft_consensus.cc:729] T 64a60f4b71464081993db3b6cc5b3eed P ebe675f1a75841f5949a6f947e8eb7f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe675f1a75841f5949a6f947e8eb7f4, State: Initialized, Role: FOLLOWER
I20260924 23:16:49.304602 11756 consensus_queue.cc:260] T 64a60f4b71464081993db3b6cc5b3eed P ebe675f1a75841f5949a6f947e8eb7f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49cfad4fe59d43f5822acd19245fb41a" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46013 } } peers { permanent_uuid: "2a1d04990b2f46f4a7e136007857d0f7" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45873 } } peers { permanent_uuid: "ebe675f1a75841f5949a6f947e8eb7f4" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42605 } }
I20260924 23:16:49.304864 11756 ts_tablet_manager.cc:1428] T 64a60f4b71464081993db3b6cc5b3eed P ebe675f1a75841f5949a6f947e8eb7f4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:16:49.305063 11470 heartbeater.cc:503] Master 127.1.4.254:33945 was elected leader, sending a full tablet report...
I20260924 23:16:49.305444 11755 raft_consensus.cc:348] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49cfad4fe59d43f5822acd19245fb41a" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46013 } } peers { permanent_uuid: "2a1d04990b2f46f4a7e136007857d0f7" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45873 } } peers { permanent_uuid: "ebe675f1a75841f5949a6f947e8eb7f4" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42605 } }
I20260924 23:16:49.305610 11755 raft_consensus.cc:374] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:49.305642 11755 raft_consensus.cc:729] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1d04990b2f46f4a7e136007857d0f7, State: Initialized, Role: FOLLOWER
I20260924 23:16:49.305747 11755 consensus_queue.cc:260] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49cfad4fe59d43f5822acd19245fb41a" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46013 } } peers { permanent_uuid: "2a1d04990b2f46f4a7e136007857d0f7" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45873 } } peers { permanent_uuid: "ebe675f1a75841f5949a6f947e8eb7f4" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42605 } }
I20260924 23:16:49.306010 11755 ts_tablet_manager.cc:1428] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:16:49.306027 11604 heartbeater.cc:503] Master 127.1.4.254:33945 was elected leader, sending a full tablet report...
I20260924 23:16:49.306026 11759 raft_consensus.cc:348] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49cfad4fe59d43f5822acd19245fb41a" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46013 } } peers { permanent_uuid: "2a1d04990b2f46f4a7e136007857d0f7" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45873 } } peers { permanent_uuid: "ebe675f1a75841f5949a6f947e8eb7f4" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42605 } }
I20260924 23:16:49.306169 11759 raft_consensus.cc:374] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:16:49.306206 11759 raft_consensus.cc:729] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49cfad4fe59d43f5822acd19245fb41a, State: Initialized, Role: FOLLOWER
I20260924 23:16:49.306452 11759 consensus_queue.cc:260] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49cfad4fe59d43f5822acd19245fb41a" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46013 } } peers { permanent_uuid: "2a1d04990b2f46f4a7e136007857d0f7" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45873 } } peers { permanent_uuid: "ebe675f1a75841f5949a6f947e8eb7f4" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42605 } }
I20260924 23:16:49.306703 11759 ts_tablet_manager.cc:1428] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:16:49.306733 11738 heartbeater.cc:503] Master 127.1.4.254:33945 was elected leader, sending a full tablet report...
I20260924 23:16:49.344162 11762 raft_consensus.cc:482] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:16:49.344285 11762 raft_consensus.cc:504] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49cfad4fe59d43f5822acd19245fb41a" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46013 } } peers { permanent_uuid: "2a1d04990b2f46f4a7e136007857d0f7" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45873 } } peers { permanent_uuid: "ebe675f1a75841f5949a6f947e8eb7f4" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42605 } }
I20260924 23:16:49.344645 11762 leader_election.cc:288] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49cfad4fe59d43f5822acd19245fb41a (127.1.4.195:46013), ebe675f1a75841f5949a6f947e8eb7f4 (127.1.4.193:42605)
I20260924 23:16:49.347448 11752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:49.344885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:44399 (local address 127.1.4.195:46013)
0924 23:16:49.344939 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:49.344942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:49.345186 (+ 244us) server_negotiation.cc:407] Connection header received
0924 23:16:49.345213 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:49.345214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:49.345227 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:49.345276 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:49.345394 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.345857 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.346754 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.347115 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.347168 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:49.347178 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:49.347304 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 23:16:49.347346 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:16:49.347464 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:49.344956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:37283 (local address 127.1.4.193:42605)
0924 23:16:49.344995 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:49.344997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:49.345005 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:49.345029 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:49.345029 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:49.345037 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:49.345080 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:16:49.345254 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.345775 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.346740 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.347128 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.347189 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:49.347198 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:49.347343 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 23:16:49.347402 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:16:49.347471 11606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:49.344817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:42605 (local address 127.1.4.194:37283)
0924 23:16:49.344865 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:49.344877 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:49.344919 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:49.345103 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:49.345104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:49.345105 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:49.345175 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:49.345177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.345855 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.345856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:49.346656 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:49.346658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.347203 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.347204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:49.347275 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:49.347278 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:49.347331 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 23:16:49.347388 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 23:16:49.347486 11764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:49.344817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:46013 (local address 127.1.4.194:44399)
0924 23:16:49.345106 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:49.345115 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:49.345179 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:49.345278 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:16:49.345279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:16:49.345280 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:16:49.345340 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:49.345341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.345926 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.345927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:49.346671 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:16:49.346673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.347182 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.347183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:16:49.347246 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:49.347249 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:16:49.347288 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 23:16:49.347320 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":49,"thread_start_us":178,"threads_started":1}
I20260924 23:16:49.347851 11424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a60f4b71464081993db3b6cc5b3eed" candidate_uuid: "2a1d04990b2f46f4a7e136007857d0f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe675f1a75841f5949a6f947e8eb7f4" is_pre_election: true
I20260924 23:16:49.347990 11692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a60f4b71464081993db3b6cc5b3eed" candidate_uuid: "2a1d04990b2f46f4a7e136007857d0f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49cfad4fe59d43f5822acd19245fb41a" is_pre_election: true
I20260924 23:16:49.348037 11424 raft_consensus.cc:2454] T 64a60f4b71464081993db3b6cc5b3eed P ebe675f1a75841f5949a6f947e8eb7f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1d04990b2f46f4a7e136007857d0f7 in term 0.
I20260924 23:16:49.348099 11692 raft_consensus.cc:2454] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a1d04990b2f46f4a7e136007857d0f7 in term 0.
I20260924 23:16:49.348253 11494 leader_election.cc:302] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a1d04990b2f46f4a7e136007857d0f7, ebe675f1a75841f5949a6f947e8eb7f4; no voters:
I20260924 23:16:49.348380 11762 raft_consensus.cc:2781] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:16:49.348443 11762 raft_consensus.cc:482] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:16:49.348467 11762 raft_consensus.cc:3037] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:49.349135 11762 raft_consensus.cc:504] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49cfad4fe59d43f5822acd19245fb41a" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46013 } } peers { permanent_uuid: "2a1d04990b2f46f4a7e136007857d0f7" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45873 } } peers { permanent_uuid: "ebe675f1a75841f5949a6f947e8eb7f4" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42605 } }
I20260924 23:16:49.349246 11762 leader_election.cc:288] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [CANDIDATE]: Term 1 election: Requested vote from peers 49cfad4fe59d43f5822acd19245fb41a (127.1.4.195:46013), ebe675f1a75841f5949a6f947e8eb7f4 (127.1.4.193:42605)
I20260924 23:16:49.349395 11692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a60f4b71464081993db3b6cc5b3eed" candidate_uuid: "2a1d04990b2f46f4a7e136007857d0f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49cfad4fe59d43f5822acd19245fb41a"
I20260924 23:16:49.349471 11692 raft_consensus.cc:3037] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:49.350175 11692 raft_consensus.cc:2454] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1d04990b2f46f4a7e136007857d0f7 in term 1.
I20260924 23:16:49.350271 11424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a60f4b71464081993db3b6cc5b3eed" candidate_uuid: "2a1d04990b2f46f4a7e136007857d0f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe675f1a75841f5949a6f947e8eb7f4"
I20260924 23:16:49.350374 11491 leader_election.cc:302] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a1d04990b2f46f4a7e136007857d0f7, 49cfad4fe59d43f5822acd19245fb41a; no voters:
I20260924 23:16:49.350364 11424 raft_consensus.cc:3037] T 64a60f4b71464081993db3b6cc5b3eed P ebe675f1a75841f5949a6f947e8eb7f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:16:49.350461 11762 raft_consensus.cc:2781] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:16:49.350617 11762 raft_consensus.cc:686] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [term 1 LEADER]: Becoming Leader. State: Replica: 2a1d04990b2f46f4a7e136007857d0f7, State: Running, Role: LEADER
I20260924 23:16:49.350714 11762 consensus_queue.cc:237] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49cfad4fe59d43f5822acd19245fb41a" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46013 } } peers { permanent_uuid: "2a1d04990b2f46f4a7e136007857d0f7" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45873 } } peers { permanent_uuid: "ebe675f1a75841f5949a6f947e8eb7f4" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42605 } }
I20260924 23:16:49.351171 11424 raft_consensus.cc:2454] T 64a60f4b71464081993db3b6cc5b3eed P ebe675f1a75841f5949a6f947e8eb7f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1d04990b2f46f4a7e136007857d0f7 in term 1.
I20260924 23:16:49.351703 11278 catalog_manager.cc:5495] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a1d04990b2f46f4a7e136007857d0f7 (127.1.4.194). New cstate: current_term: 1 leader_uuid: "2a1d04990b2f46f4a7e136007857d0f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49cfad4fe59d43f5822acd19245fb41a" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1d04990b2f46f4a7e136007857d0f7" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebe675f1a75841f5949a6f947e8eb7f4" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42605 } health_report { overall_health: UNKNOWN } } }
I20260924 23:16:49.367612 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:16:49.365256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55104 (local address 127.1.4.194:45873)
0924 23:16:49.365354 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 23:16:49.365356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:16:49.365364 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:16:49.365398 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:16:49.365399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:16:49.365419 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:16:49.365481 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:16:49.365599 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.366065 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.366811 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:16:49.367000 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:16:49.367056 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:16:49.367190 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:16:49.367297 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:16:49.367411 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 23:16:49.367481 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 23:16:49.367511 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 23:16:49.367635 11605 tablet.cc:2369] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:16:49.369529 11692 raft_consensus.cc:1261] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a [term 1 FOLLOWER]: Refusing update from remote peer 2a1d04990b2f46f4a7e136007857d0f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:49.369529 11424 raft_consensus.cc:1261] T 64a60f4b71464081993db3b6cc5b3eed P ebe675f1a75841f5949a6f947e8eb7f4 [term 1 FOLLOWER]: Refusing update from remote peer 2a1d04990b2f46f4a7e136007857d0f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:16:49.369913 11762 consensus_queue.cc:1035] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49cfad4fe59d43f5822acd19245fb41a" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:49.370076 11765 consensus_queue.cc:1035] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebe675f1a75841f5949a6f947e8eb7f4" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 42605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:16:49.374660 11278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:49.374752 11278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:16:49.375541 11278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fba529caab2442d8af83ac54d86fc28d: Sending DeleteTablet for 3 replicas of tablet 64a60f4b71464081993db3b6cc5b3eed
I20260924 23:16:49.375836 11538 tablet_service.cc:1514] Processing DeleteTablet for tablet 64a60f4b71464081993db3b6cc5b3eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:49 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:55092
I20260924 23:16:49.375880 11404 tablet_service.cc:1514] Processing DeleteTablet for tablet 64a60f4b71464081993db3b6cc5b3eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:49 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:38716
I20260924 23:16:49.376050 11777 tablet_replica.cc:330] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7: stopping tablet replica
I20260924 23:16:49.376029 11672 tablet_service.cc:1514] Processing DeleteTablet for tablet 64a60f4b71464081993db3b6cc5b3eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:16:49 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:35188
I20260924 23:16:49.376132 11777 raft_consensus.cc:2229] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:16:49.376224 11777 raft_consensus.cc:2258] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:49.376510 11779 tablet_replica.cc:330] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a: stopping tablet replica
I20260924 23:16:49.376564 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 11341
I20260924 23:16:49.376603 11779 raft_consensus.cc:2229] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:16:49.376693 11779 raft_consensus.cc:2258] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:16:49.377060 11779 ts_tablet_manager.cc:1905] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:49.378731 11777 ts_tablet_manager.cc:1905] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:16:49.378925 11779 ts_tablet_manager.cc:1918] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:49.378986 11779 log.cc:1196] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/64a60f4b71464081993db3b6cc5b3eed
I20260924 23:16:49.379220 11779 ts_tablet_manager.cc:1939] T 64a60f4b71464081993db3b6cc5b3eed P 49cfad4fe59d43f5822acd19245fb41a: Deleting consensus metadata
I20260924 23:16:49.379624 11263 catalog_manager.cc:4841] TS 49cfad4fe59d43f5822acd19245fb41a (127.1.4.195:46013): tablet 64a60f4b71464081993db3b6cc5b3eed (table test-table [id=deef3cc87c6b466db987fc0e2f801cd2]) successfully deleted
I20260924 23:16:49.380169 11777 ts_tablet_manager.cc:1918] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:16:49.380214 11777 log.cc:1196] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/64a60f4b71464081993db3b6cc5b3eed
I20260924 23:16:49.380379 11777 ts_tablet_manager.cc:1939] T 64a60f4b71464081993db3b6cc5b3eed P 2a1d04990b2f46f4a7e136007857d0f7: Deleting consensus metadata
I20260924 23:16:49.381306 11264 catalog_manager.cc:4841] TS 2a1d04990b2f46f4a7e136007857d0f7 (127.1.4.194:45873): tablet 64a60f4b71464081993db3b6cc5b3eed (table test-table [id=deef3cc87c6b466db987fc0e2f801cd2]) successfully deleted
W20260924 23:16:49.382560 11266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.4.193:42605 (error 108)
W20260924 23:16:49.383391 11749 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:16:49.383035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:42605 (local address 127.0.0.1:38726)
0924 23:16:49.383090 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 23:16:49.383101 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:16:49.383139 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:16:49.383321 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:42605: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 23:16:49.383531 11266 catalog_manager.cc:4584] TS ebe675f1a75841f5949a6f947e8eb7f4 (127.1.4.193:42605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 64a60f4b71464081993db3b6cc5b3eed: Network error: Client connection negotiation failed: client connection to 127.1.4.193:42605: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 23:16:49.383771 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 11475
I20260924 23:16:49.387866 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 11609
I20260924 23:16:49.391819 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 11247
2026-09-24T23:16:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1831 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:16:49 dist-test-slave-s1hw krb5kdc[11783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:16:49 dist-test-slave-s1hw krb5kdc[11783](info): set up 2 sockets
Sep 24 23:16:49 dist-test-slave-s1hw krb5kdc[11783](info): commencing operation
krb5kdc: starting...
W20260924 23:16:50.348009 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:16:50 dist-test-slave-s1hw krb5kdc[11783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:16:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:16:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.4.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.4.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.4.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/rangeradmin_127.1.4.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.4.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/rangeradmin_127.1.4.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.4.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.4.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.4.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/rangerlookup_127.1.4.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.4.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/rangerlookup_127.1.4.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.4.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.4.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.4.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/HTTP_127.1.4.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.4.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/HTTP_127.1.4.212@KRBTEST.COM.keytab.
I20260924 23:16:50.510406 1043 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskEEvosl/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/postgres -l logfile start
2026-09-24 23:16:53.045 UTC [11813] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 23:16:53.045 UTC [11813] LOG: listening on IPv4 address "127.1.4.212", port 51261
2026-09-24 23:16:53.049 UTC [11813] LOG: listening on Unix socket "/tmp/.s.PGSQL.51261"
2026-09-24 23:16:53.072 UTC [11816] LOG: database system was shut down at 2026-09-24 23:16:51 UTC
2026-09-24 23:16:53.076 UTC [11813] LOG: database system is ready to accept connections
I20260924 23:16:53.947937 1043 mini_postgres.cc:96] Postgres bound to 51261
2026-09-24 23:16:53.953 UTC [11824] FATAL: database "slave" does not exist
127.1.4.212:51261 - accepting connections
I20260924 23:16:53.954457 1043 mini_ranger.cc:161] Starting Ranger...
I20260924 23:16:53.969918 1043 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 23:16:54.485743 1043 mini_ranger.cc:87] Created ranger Postgres database
I20260924 23:16:54.485831 1043 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-admin
2026-09-24 23:16:54,757 [I] DB FLAVOR :POSTGRES
2026-09-24 23:16:54,757 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 23:16:54,757 [I] Checking connection..
2026-09-24 23:16:54,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 23:16:55,022 [I] Checking connection passed.
2026-09-24 23:16:55,022 [I] --------- Verifying version history table ---------
2026-09-24 23:16:55,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 23:16:55,288 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 23:16:55,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 23:16:55,525 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 23:16:55,526 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 23:16:55,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 23:16:55.771 UTC [11888] WARNING: there is no transaction in progress
2026-09-24 23:16:55,777 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 23:16:55,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 23:16:56,014 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 23:16:56,014 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 23:16:56,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 23:16:56,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 23:16:56,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s1hw.c.gcp-upstream.internal','N') ;"
2026-09-24 23:16:56,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 23:16:57,023 [I] Table x_portal_user does not exist in database ranger
2026-09-24 23:16:57,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 23:16:57,282 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 23:16:57,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 23:16:57,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 23:16:57,757 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 23:16:57,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 23:16:58.767 UTC [11996] WARNING: there is no transaction in progress
2026-09-24 23:16:58.785 UTC [11996] WARNING: there is no transaction in progress
2026-09-24 23:16:58.802 UTC [11996] WARNING: there is no transaction in progress
2026-09-24 23:16:58.818 UTC [11996] WARNING: there is no transaction in progress
2026-09-24 23:16:58.835 UTC [11996] WARNING: there is no transaction in progress
2026-09-24 23:16:58.853 UTC [11996] WARNING: there is no transaction in progress
2026-09-24 23:16:58.919 UTC [11996] WARNING: there is no transaction in progress
2026-09-24 23:16:58.926 UTC [11996] WARNING: there is no transaction in progress
2026-09-24 23:16:58.942 UTC [11996] WARNING: there is no transaction in progress
2026-09-24 23:16:58.954 UTC [11996] WARNING: there is no transaction in progress
2026-09-24 23:16:58.965 UTC [11996] WARNING: there is no transaction in progress
2026-09-24 23:16:58.979 UTC [11996] WARNING: there is no transaction in progress
2026-09-24 23:16:58.989 UTC [11996] WARNING: there is no transaction in progress
2026-09-24 23:16:58.996 UTC [11996] WARNING: there is no transaction in progress
2026-09-24 23:16:59.004 UTC [11996] WARNING: there is no transaction in progress
2026-09-24 23:16:59.012 UTC [11996] WARNING: there is no transaction in progress
2026-09-24 23:16:59,700 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 23:16:59,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 23:16:59,950 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 23:16:59,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 23:17:00,212 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 23:17:00,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 23:17:00,472 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 23:17:00,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 23:17:00,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 23:17:00,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s1hw.c.gcp-upstream.internal';"
2026-09-24 23:17:01,211 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 23:17:01,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 23:17:01,475 [I] DB_PATCHES have already been applied
I20260924 23:17:01.481436 1043 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-admin:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskEEvosl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 23:17:01.481514 1043 mini_ranger.cc:193] Using host: 127.1.4.212
I20260924 23:17:01.484568 1043 mini_ranger.cc:227] Ranger admin URL: http://127.1.4.212:44409
Sep 24, 2026 11:17:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:17:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:17:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 11:17:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 11:17:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 11:17:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 11:17:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 23:17:02.500772 1043 mini_ranger.cc:160] Time spent starting Ranger: real 8.546s user 0.000s sys 0.009s
Sep 24, 2026 11:17:22 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 11:17:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 11:17:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 11:17:22 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 11:17:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 11:17:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 11:17:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 11:17:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 11:17:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 11:17:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 11:17:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 11:17:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 11:17:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 11:17:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 11:17:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 11:17:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 11:17:22 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 11:17:23 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 23:17:24.260448 1043 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.4.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.4.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.4.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/rangerkms_127.1.4.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.4.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/rangerkms_127.1.4.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.4.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.4.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.4.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/HTTP_127.1.4.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.4.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/HTTP_127.1.4.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 23:17:24 dist-test-slave-s1hw krb5kdc[11783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291844, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 23:17:24.339886 1043 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 23:17:24.349247 1043 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 23:17:25.238957 1043 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 23:17:25.239041 1043 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-kms
I20260924 23:17:25.239058 1043 mini_ranger_kms.cc:227] Using postgres at 127.1.4.212:51261
2026-09-24 23:17:25,397 [I] DB FLAVOR :POSTGRES
2026-09-24 23:17:25,397 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 23:17:25,397 [I] Checking connection
2026-09-24 23:17:25,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 23:17:25,641 [I] connection success
2026-09-24 23:17:25,642 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 23:17:25,642 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 23:17:25,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 23:17:25,884 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 23:17:25,884 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 23:17:25,885 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 23:17:25,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:51261/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 23:17:26,134 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 23:17:26.597198 1043 mini_ranger_kms.cc:310] Created kms service
I20260924 23:17:26.817108 1043 mini_ranger_kms.cc:326] Created kudu user
I20260924 23:17:26.858769 1043 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 23:17:26 dist-test-slave-s1hw krb5kdc[11783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291844, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:17:27.228060 1043 mini_ranger_kms.cc:384] Added ranger policy
I20260924 23:17:27.228181 1043 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-kms:/tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 23:17:27.228207 1043 mini_ranger_kms.cc:240] Using host: 127.1.4.212
I20260924 23:17:27.231177 1043 mini_ranger_kms.cc:276] Ranger KMS PID: 12176
I20260924 23:17:27.231294 1043 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.4.212:48171
Sep 24, 2026 11:17:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:17:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:17:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 11:17:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 11:17:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 11:17:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 23:17:29.270893 1043 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.931s user 0.021s sys 0.120s
I20260924 23:17:29.270998 1043 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 23:17:29.271075 1043 mini_ranger_kms.cc:401] 127.1.4.212:48171/kms/v1/keys
Sep 24 23:17:29 dist-test-slave-s1hw krb5kdc[11783](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790291849, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.4.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 23:17:29 dist-test-slave-s1hw krb5kdc[11783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291849, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.4.212@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
Sep 24 23:17:30 dist-test-slave-s1hw krb5kdc[11783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291849, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.4.212@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
Sep 24 23:17:32 dist-test-slave-s1hw krb5kdc[11783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:17:32.443940 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:39315
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:39315
--ranger_config_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.4.212:48171/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:17:32.525508 12227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:17:32.525714 12227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:17:32.525734 12227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:17:32.526885 12227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:17:32.526926 12227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:17:32.526938 12227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:17:32.526957 12227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:17:32.526980 12227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:17:32.528527 12227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.4.212:48171/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:39315
--ranger_config_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:39315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:17:32.528752 12227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:17:32.528961 12227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:17:32.534308 12235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:17:32.534444 12227 server_base.cc:1034] running on GCE node
W20260924 23:17:32.534371 12232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:17:32.534308 12233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:17:32.534799 12227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:17:32.535104 12227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:17:32.536242 12227 hybrid_clock.cc:648] HybridClock initialized: now 1790291852536222 us; error 35 us; skew 500 ppm
Sep 24 23:17:32 dist-test-slave-s1hw krb5kdc[11783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291852, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:17:32.538748 12227 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:17:32.539188 12227 webserver.cc:460] Webserver started at http://127.1.4.254:38975/ using document root <none> and password file <none>
I20260924 23:17:32.539382 12227 fs_manager.cc:359] Metadata directory not provided
I20260924 23:17:32.539429 12227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:17:32.539525 12227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 23:17:32 dist-test-slave-s1hw krb5kdc[11783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291852, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:17:32.591740 12227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "c3122b59aaf546c0a43f97f7462a52ee"
format_stamp: "Formatted at 2026-09-24 23:17:32 on dist-test-slave-s1hw"
server_key: "378872171d951b3e083a408b744559db"
server_key_iv: "20c1dd6cf8a6c51d43a2ffaa0f8d5f6f"
server_key_version: "kuduclusterkey@0"
I20260924 23:17:32.592106 12227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "c3122b59aaf546c0a43f97f7462a52ee"
format_stamp: "Formatted at 2026-09-24 23:17:32 on dist-test-slave-s1hw"
server_key: "378872171d951b3e083a408b744559db"
server_key_iv: "20c1dd6cf8a6c51d43a2ffaa0f8d5f6f"
server_key_version: "kuduclusterkey@0"
I20260924 23:17:32.593466 12227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:17:32.605223 12242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:17:32.605406 12227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:17:32.605504 12227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "c3122b59aaf546c0a43f97f7462a52ee"
format_stamp: "Formatted at 2026-09-24 23:17:32 on dist-test-slave-s1hw"
server_key: "378872171d951b3e083a408b744559db"
server_key_iv: "20c1dd6cf8a6c51d43a2ffaa0f8d5f6f"
server_key_version: "kuduclusterkey@0"
I20260924 23:17:32.605600 12227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:17:32.624164 12227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:17:32.626211 12227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:17:32.626343 12227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:17:32.631390 12227 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:39315
I20260924 23:17:32.631436 12294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:39315 every 8 connection(s)
I20260924 23:17:32.631852 12227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:17:32.632495 12295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:17:32.634682 12295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee: Bootstrap starting.
I20260924 23:17:32.635236 12295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee: Neither blocks nor log segments found. Creating new log.
I20260924 23:17:32.635519 12295 log.cc:824] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee: Log is configured to *not* fsync() on all Append() calls
I20260924 23:17:32.636154 12295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee: No bootstrap required, opened a new log
I20260924 23:17:32.638072 12295 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3122b59aaf546c0a43f97f7462a52ee" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 39315 } }
I20260924 23:17:32.638211 12295 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:17:32.638242 12295 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3122b59aaf546c0a43f97f7462a52ee, State: Initialized, Role: FOLLOWER
I20260924 23:17:32.638387 12295 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3122b59aaf546c0a43f97f7462a52ee" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 39315 } }
I20260924 23:17:32.638465 12295 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:17:32.638496 12295 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:17:32.638542 12295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:17:32.639051 12295 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3122b59aaf546c0a43f97f7462a52ee" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 39315 } }
I20260924 23:17:32.639142 12295 leader_election.cc:302] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3122b59aaf546c0a43f97f7462a52ee; no voters:
I20260924 23:17:32.639298 12295 leader_election.cc:288] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:17:32.639346 12298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:17:32.639528 12298 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [term 1 LEADER]: Becoming Leader. State: Replica: c3122b59aaf546c0a43f97f7462a52ee, State: Running, Role: LEADER
I20260924 23:17:32.639564 12295 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:17:32.639707 12298 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3122b59aaf546c0a43f97f7462a52ee" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 39315 } }
I20260924 23:17:32.639818 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 12227
I20260924 23:17:32.639914 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:17:32.640106 1043 external_mini_cluster.cc:1428] Setting key 1da2583d37bf311422106aa15e6f73f1
I20260924 23:17:32.640339 12300 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3122b59aaf546c0a43f97f7462a52ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3122b59aaf546c0a43f97f7462a52ee" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 39315 } } }
I20260924 23:17:32.640426 12300 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [sys.catalog]: This master's current role is: LEADER
I20260924 23:17:32.641278 12299 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3122b59aaf546c0a43f97f7462a52ee. Latest consensus state: current_term: 1 leader_uuid: "c3122b59aaf546c0a43f97f7462a52ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3122b59aaf546c0a43f97f7462a52ee" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 39315 } } }
I20260924 23:17:32.641422 12299 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee [sys.catalog]: This master's current role is: LEADER
I20260924 23:17:32.641593 12309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:17:32.641672 12295 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 23:17:32.642069 12309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:17:32.645197 12309 catalog_manager.cc:1347] Generated new cluster ID: 134d01d53edf4f6bbe83491889d7d515
I20260924 23:17:32.645290 12309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:17:32 dist-test-slave-s1hw krb5kdc[11783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:17:32.653422 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:32.640748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.1.4.254:39315)
0924 23:17:32.641157 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0924 23:17:32.641168 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:17:32.641205 (+ 37us) server_negotiation.cc:407] Connection header received
0924 23:17:32.641328 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:17:32.641335 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:17:32.641398 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:17:32.643550 (+ 2152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:17:32.643647 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:32.645062 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:32.646109 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:32.646713 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:32.648429 (+ 1716us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:32.649444 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:17:32.649447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:17:32.649448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:17:32.649459 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:17:32.651352 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:17:32.651763 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:17:32.651766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:17:32.651767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:17:32.651875 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:17:32.652083 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:17:32.652083 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:17:32.652084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:17:32.652419 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:17:32.652997 (+ 578us) server_negotiation.cc:1035] Waiting for connection context
0924 23:17:32.653071 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 23:17:32.653190 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":236,"threads_started":1}
I20260924 23:17:32.670976 12309 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:17:32.672199 12309 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:17:32.679230 12309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee: Generated new TSK 0
I20260924 23:17:32.679530 12309 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:17:33.406170 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.4.212:48171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:39315
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:17:33.520740 12340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:17:33.520951 12340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:17:33.520982 12340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:17:33.522161 12340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:17:33.522203 12340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:17:33.522239 12340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:17:33.523806 12340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.4.212:48171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:39315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:17:33.524031 12340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:17:33.524236 12340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:17:33.530071 12348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:17:33.530169 12346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:17:33.530066 12345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:17:33.530354 12340 server_base.cc:1034] running on GCE node
I20260924 23:17:33.530597 12340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:17:33.530846 12340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:17:33.531972 12340 hybrid_clock.cc:648] HybridClock initialized: now 1790291853531960 us; error 25 us; skew 500 ppm
Sep 24 23:17:33 dist-test-slave-s1hw krb5kdc[11783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:17:33.534374 12340 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:17:33.534818 12340 webserver.cc:460] Webserver started at http://127.1.4.193:45223/ using document root <none> and password file <none>
I20260924 23:17:33.535027 12340 fs_manager.cc:359] Metadata directory not provided
I20260924 23:17:33.535072 12340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:17:33.535200 12340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 23:17:33 dist-test-slave-s1hw krb5kdc[11783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:17:33.548795 12340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "3ad1a0bcadfb4063ad2ee5d527719062"
format_stamp: "Formatted at 2026-09-24 23:17:33 on dist-test-slave-s1hw"
server_key: "be9837095b2f2d1efb01e5090b0cbb93"
server_key_iv: "3875dfd73233fe211bbbc95f8e1d2f84"
server_key_version: "kuduclusterkey@0"
I20260924 23:17:33.549315 12340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "3ad1a0bcadfb4063ad2ee5d527719062"
format_stamp: "Formatted at 2026-09-24 23:17:33 on dist-test-slave-s1hw"
server_key: "be9837095b2f2d1efb01e5090b0cbb93"
server_key_iv: "3875dfd73233fe211bbbc95f8e1d2f84"
server_key_version: "kuduclusterkey@0"
I20260924 23:17:33.555763 12340 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260924 23:17:33.570962 12357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:17:33.571149 12340 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260924 23:17:33.571244 12340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "3ad1a0bcadfb4063ad2ee5d527719062"
format_stamp: "Formatted at 2026-09-24 23:17:33 on dist-test-slave-s1hw"
server_key: "be9837095b2f2d1efb01e5090b0cbb93"
server_key_iv: "3875dfd73233fe211bbbc95f8e1d2f84"
server_key_version: "kuduclusterkey@0"
I20260924 23:17:33.571352 12340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:17:33.595534 12340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:17:33.597941 12340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:17:33.598093 12340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:17:33.598320 12340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:17:33.598683 12340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:17:33.598730 12340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:17:33.598760 12340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:17:33.598827 12340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:17:33.606061 12340 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:37063
I20260924 23:17:33.606509 12340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:17:33.606824 12470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:37063 every 8 connection(s)
I20260924 23:17:33.613118 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 12340
I20260924 23:17:33.613229 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
Sep 24 23:17:33 dist-test-slave-s1hw krb5kdc[11783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:17:33.613418 1043 external_mini_cluster.cc:1428] Setting key 94b21d2371050734d12bcf23212691b9
I20260924 23:17:33.617075 12473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:33.607252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:39315 (local address 127.1.4.193:46743)
0924 23:17:33.607441 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 23:17:33.607482 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:17:33.607952 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:17:33.608334 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:17:33.608340 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:17:33.608591 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:17:33.608931 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:17:33.608939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:17:33.609757 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:17:33.609758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:17:33.611061 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:17:33.611063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:17:33.611473 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:17:33.611474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:17:33.611614 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:33.612181 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:17:33.612192 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:17:33.613672 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:17:33.615389 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:17:33.615396 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:17:33.615398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:17:33.615692 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:17:33.615904 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:17:33.615905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:17:33.615905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:17:33.616031 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:17:33.616289 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:17:33.616297 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:17:33.616485 (+ 188us) client_negotiation.cc:769] Sending connection context
0924 23:17:33.616604 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 23:17:33.616768 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260924 23:17:33.617501 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:33.607280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:46743 (local address 127.1.4.254:39315)
0924 23:17:33.608113 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0924 23:17:33.608115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:17:33.608121 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:17:33.608142 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:17:33.608143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:17:33.608153 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:17:33.608204 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:17:33.609080 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:33.609637 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:33.611160 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:33.611392 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:33.611449 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:33.613754 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:17:33.613755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:17:33.613756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:17:33.613764 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:17:33.615294 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:17:33.615776 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:17:33.615777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:17:33.615778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:17:33.615842 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:17:33.616088 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:17:33.616089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:17:33.616089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:17:33.616218 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:17:33.616277 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 23:17:33.617315 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0924 23:17:33.617404 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
I20260924 23:17:33.623821 12471 heartbeater.cc:347] Connected to a master server at 127.1.4.254:39315
I20260924 23:17:33.623975 12471 heartbeater.cc:464] Registering TS with master...
I20260924 23:17:33.624229 12471 heartbeater.cc:511] Master 127.1.4.254:39315 requested a full tablet report, sending...
I20260924 23:17:33.624799 12259 ts_manager.cc:194] Registered new tserver with Master: 3ad1a0bcadfb4063ad2ee5d527719062 (127.1.4.193:37063)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
I20260924 23:17:33.625707 12259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:46743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:17:33.667305 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.4.212:48171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:39315
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 23:17:33 dist-test-slave-s1hw krb5kdc[11783](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790291853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 23:17:33.761802 12479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:17:33.762217 12479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:17:33.762312 12479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:17:33.764300 12479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:17:33.764436 12479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:17:33.764529 12479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:17:33.766710 12479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.4.212:48171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:39315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:17:33.766986 12479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:17:33.767306 12479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:17:33.773744 12484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:17:33.774088 12487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:17:33.774780 12485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:17:33.775039 12479 server_base.cc:1034] running on GCE node
I20260924 23:17:33.775249 12479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:17:33.775532 12479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:17:33.778767 12479 hybrid_clock.cc:648] HybridClock initialized: now 1790291853778746 us; error 57 us; skew 500 ppm
Sep 24 23:17:33 dist-test-slave-s1hw krb5kdc[11783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:17:33.782172 12479 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:17:33.782822 12479 webserver.cc:460] Webserver started at http://127.1.4.194:32895/ using document root <none> and password file <none>
I20260924 23:17:33.783103 12479 fs_manager.cc:359] Metadata directory not provided
I20260924 23:17:33.783169 12479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:17:33.783313 12479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 23:17:33 dist-test-slave-s1hw krb5kdc[11783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:17:33.795874 12479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "f7fd68766c9c40888bae8b4911403279"
format_stamp: "Formatted at 2026-09-24 23:17:33 on dist-test-slave-s1hw"
server_key: "0c56a508a9f41d22af68f82957fc38ea"
server_key_iv: "544146a50d26287679cd0ec81bfe8e48"
server_key_version: "kuduclusterkey@0"
I20260924 23:17:33.796533 12479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "f7fd68766c9c40888bae8b4911403279"
format_stamp: "Formatted at 2026-09-24 23:17:33 on dist-test-slave-s1hw"
server_key: "0c56a508a9f41d22af68f82957fc38ea"
server_key_iv: "544146a50d26287679cd0ec81bfe8e48"
server_key_version: "kuduclusterkey@0"
I20260924 23:17:33.802619 12479 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260924 23:17:33.827725 12494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:17:33.827941 12479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 23:17:33.828048 12479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "f7fd68766c9c40888bae8b4911403279"
format_stamp: "Formatted at 2026-09-24 23:17:33 on dist-test-slave-s1hw"
server_key: "0c56a508a9f41d22af68f82957fc38ea"
server_key_iv: "544146a50d26287679cd0ec81bfe8e48"
server_key_version: "kuduclusterkey@0"
I20260924 23:17:33.828159 12479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:17:33.842491 12479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:17:33.844955 12479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:17:33.845112 12479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:17:33.845381 12479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:17:33.845739 12479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:17:33.845782 12479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:17:33.845813 12479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:17:33.845855 12479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:17:33.853336 12479 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:44565
I20260924 23:17:33.853758 12479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:17:33.857482 12608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:44565 every 8 connection(s)
I20260924 23:17:33.860421 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 12479
I20260924 23:17:33.860536 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:17:33.860713 1043 external_mini_cluster.cc:1428] Setting key 267c8f2283de37088542d2037dd612c0
Sep 24 23:17:33 dist-test-slave-s1hw krb5kdc[11783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:17:33.867092 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:33.858088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:49467 (local address 127.1.4.254:39315)
0924 23:17:33.858140 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 23:17:33.858142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:17:33.858373 (+ 231us) server_negotiation.cc:407] Connection header received
0924 23:17:33.858903 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:17:33.858904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:17:33.858917 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:17:33.858967 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:17:33.859569 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:33.860109 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:33.861156 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:33.861412 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:33.861466 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:33.863751 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:17:33.863753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:17:33.863753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:17:33.863761 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:17:33.865419 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:17:33.865939 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:17:33.865941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:17:33.865941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:17:33.866010 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:17:33.866307 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:17:33.866309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:17:33.866309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:17:33.866473 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:17:33.866547 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 23:17:33.866902 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 23:17:33.866995 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:17:33.867274 12611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:33.858084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:39315 (local address 127.1.4.194:49467)
0924 23:17:33.858240 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 23:17:33.858284 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:17:33.858774 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:17:33.859045 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:17:33.859050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:17:33.859233 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:17:33.859487 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:17:33.859492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:17:33.860198 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:17:33.860199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:17:33.861075 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:17:33.861078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:17:33.861538 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:17:33.861540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:17:33.861691 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:33.862281 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:17:33.862293 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:17:33.863653 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:17:33.865516 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:17:33.865525 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:17:33.865527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:17:33.865838 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:17:33.866083 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:17:33.866084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:17:33.866085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:17:33.866229 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:17:33.866544 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:17:33.866550 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:17:33.866742 (+ 192us) client_negotiation.cc:769] Sending connection context
0924 23:17:33.866873 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 23:17:33.867041 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260924 23:17:33.867746 12609 heartbeater.cc:347] Connected to a master server at 127.1.4.254:39315
I20260924 23:17:33.867854 12609 heartbeater.cc:464] Registering TS with master...
I20260924 23:17:33.868052 12609 heartbeater.cc:511] Master 127.1.4.254:39315 requested a full tablet report, sending...
I20260924 23:17:33.868455 12259 ts_manager.cc:194] Registered new tserver with Master: f7fd68766c9c40888bae8b4911403279 (127.1.4.194:44565)
I20260924 23:17:33.868956 12259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:49467
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:17:33.889922 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.4.212:48171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:39315
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:17:34.007056 12614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:17:34.007265 12614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:17:34.007289 12614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:17:34.008574 12614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:17:34.008627 12614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:17:34.008665 12614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:17:34.010519 12614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.4.212:48171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:39315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:17:34.010740 12614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:17:34.010978 12614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:17:34.016201 12619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:17:34.016208 12622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:17:34.016538 12614 server_base.cc:1034] running on GCE node
W20260924 23:17:34.016211 12620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:17:34.016835 12614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:17:34.017112 12614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:17:34.018286 12614 hybrid_clock.cc:648] HybridClock initialized: now 1790291854018273 us; error 26 us; skew 500 ppm
Sep 24 23:17:34 dist-test-slave-s1hw krb5kdc[11783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291854, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:17:34.021489 12614 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:17:34.022157 12614 webserver.cc:460] Webserver started at http://127.1.4.195:45041/ using document root <none> and password file <none>
I20260924 23:17:34.022434 12614 fs_manager.cc:359] Metadata directory not provided
I20260924 23:17:34.022500 12614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:17:34.022643 12614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 23:17:34 dist-test-slave-s1hw krb5kdc[11783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291854, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:17:34.050571 12614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "95ad0ce3371a4c05a38e3199cc32326e"
format_stamp: "Formatted at 2026-09-24 23:17:34 on dist-test-slave-s1hw"
server_key: "2a6d8e7a186a48f9f3608f5c8dc3f025"
server_key_iv: "0dbd32d9a5ef9cf132285137ebc6b0a4"
server_key_version: "kuduclusterkey@0"
I20260924 23:17:34.051064 12614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "95ad0ce3371a4c05a38e3199cc32326e"
format_stamp: "Formatted at 2026-09-24 23:17:34 on dist-test-slave-s1hw"
server_key: "2a6d8e7a186a48f9f3608f5c8dc3f025"
server_key_iv: "0dbd32d9a5ef9cf132285137ebc6b0a4"
server_key_version: "kuduclusterkey@0"
I20260924 23:17:34.052908 12614 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 23:17:34.082477 12629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:17:34.085739 12614 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.001s sys 0.000s
I20260924 23:17:34.085909 12614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "95ad0ce3371a4c05a38e3199cc32326e"
format_stamp: "Formatted at 2026-09-24 23:17:34 on dist-test-slave-s1hw"
server_key: "2a6d8e7a186a48f9f3608f5c8dc3f025"
server_key_iv: "0dbd32d9a5ef9cf132285137ebc6b0a4"
server_key_version: "kuduclusterkey@0"
I20260924 23:17:34.086028 12614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:17:34.108758 12614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:17:34.111258 12614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:17:34.111493 12614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:17:34.111769 12614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:17:34.112140 12614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:17:34.112191 12614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:17:34.112226 12614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:17:34.112243 12614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:17:34.119155 12614 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:40309
I20260924 23:17:34.119663 12614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:17:34.121783 12742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:40309 every 8 connection(s)
I20260924 23:17:34.127261 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 12614
I20260924 23:17:34.127475 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:17:34.127666 1043 external_mini_cluster.cc:1428] Setting key 0047a450324062d3d94aa576a7e9da0f
Sep 24 23:17:34 dist-test-slave-s1hw krb5kdc[11783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291854, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:17:34.135089 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:34.122860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:35783 (local address 127.1.4.254:39315)
0924 23:17:34.122904 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:17:34.122906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:17:34.123220 (+ 314us) server_negotiation.cc:407] Connection header received
0924 23:17:34.123952 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:17:34.123954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:17:34.123967 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:17:34.123998 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:17:34.124850 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.125637 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.127110 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.128398 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.128450 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:34.131597 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:17:34.131599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:17:34.131599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:17:34.131608 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:17:34.133347 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:17:34.133888 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:17:34.133889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:17:34.133890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:17:34.133954 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:17:34.134260 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:17:34.134261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:17:34.134261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:17:34.134425 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:17:34.134473 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 23:17:34.134896 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 23:17:34.134990 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:17:34.136209 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:34.122582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:39315 (local address 127.1.4.195:35783)
0924 23:17:34.123089 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 23:17:34.123133 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:17:34.123823 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:17:34.124187 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:17:34.124192 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:17:34.124454 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:17:34.124734 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:17:34.124741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.125749 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.125750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:17:34.127010 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:17:34.127013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.128552 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.128553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:17:34.128875 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:34.129703 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:17:34.129715 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:17:34.131464 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:17:34.133444 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:17:34.133453 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:17:34.133455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:17:34.133789 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:17:34.134027 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:17:34.134028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:17:34.134029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:17:34.134177 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:17:34.134522 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:17:34.134529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:17:34.134722 (+ 193us) client_negotiation.cc:769] Sending connection context
0924 23:17:34.135232 (+ 510us) client_negotiation.cc:240] Negotiation successful
0924 23:17:34.135481 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":77,"threads_started":1}
I20260924 23:17:34.137650 12743 heartbeater.cc:347] Connected to a master server at 127.1.4.254:39315
I20260924 23:17:34.137760 12743 heartbeater.cc:464] Registering TS with master...
I20260924 23:17:34.137966 12743 heartbeater.cc:511] Master 127.1.4.254:39315 requested a full tablet report, sending...
I20260924 23:17:34.138340 12259 ts_manager.cc:194] Registered new tserver with Master: 95ad0ce3371a4c05a38e3199cc32326e (127.1.4.195:40309)
I20260924 23:17:34.138839 12259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:35783
I20260924 23:17:34.143982 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:17:34.154285 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:34.145840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.1.4.254:39315)
0924 23:17:34.145890 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:17:34.145893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:17:34.146072 (+ 179us) server_negotiation.cc:407] Connection header received
0924 23:17:34.146146 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:17:34.146147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:17:34.146159 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:17:34.146228 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:17:34.147037 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.147846 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.149009 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.149328 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.149384 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:34.151150 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:17:34.151152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:17:34.151152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:17:34.151160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:17:34.152944 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:17:34.153350 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:17:34.153352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:17:34.153352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:17:34.153416 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:17:34.153623 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:17:34.153624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:17:34.153625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:17:34.153776 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:17:34.153857 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 23:17:34.154105 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 23:17:34.154186 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:17:34.156559 12259 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:17:34.158808 12259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:17:34.167716 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:34.162285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:44565 (local address 127.0.0.1:49220)
0924 23:17:34.162896 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0924 23:17:34.162941 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 23:17:34.163004 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:17:34.163669 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:17:34.163669 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:17:34.163671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:17:34.163738 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:17:34.163740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.165060 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.165061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:17:34.166432 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:17:34.166433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.167419 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.167420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:17:34.167499 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:34.167515 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 23:17:34.167573 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 23:17:34.167629 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":48,"thread_start_us":70,"threads_started":1}
I20260924 23:17:34.168107 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:34.162479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.1.4.194:44565)
0924 23:17:34.163472 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0924 23:17:34.163476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:17:34.163485 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:17:34.163523 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:17:34.163525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:17:34.163538 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:17:34.163599 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:17:34.163947 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.164982 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.166538 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.167264 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.167321 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:34.167965 (+ 644us) server_negotiation.cc:1035] Waiting for connection context
0924 23:17:34.167991 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 23:17:34.168029 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":958,"thread_start_us":96,"threads_started":1}
I20260924 23:17:34.168264 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:34.162665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:37063 (local address 127.0.0.1:48260)
0924 23:17:34.163269 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0924 23:17:34.163278 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:17:34.163326 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:17:34.163654 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:17:34.163657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:17:34.163660 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:17:34.163738 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:17:34.163741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.165324 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.165325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:17:34.166430 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:17:34.166432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.168038 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.168039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:17:34.168102 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:34.168106 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:17:34.168157 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 23:17:34.168198 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":2,"thread_start_us":63,"threads_started":1}
I20260924 23:17:34.168483 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:34.162754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48260 (local address 127.1.4.193:37063)
0924 23:17:34.163214 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0924 23:17:34.163219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:17:34.163323 (+ 104us) server_negotiation.cc:407] Connection header received
0924 23:17:34.163466 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:17:34.163469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:17:34.163487 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:17:34.163565 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:17:34.163816 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.165252 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.166519 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.167637 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.167693 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:34.167749 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:17:34.168353 (+ 604us) server_negotiation.cc:299] Negotiation successful
0924 23:17:34.168416 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":41,"threads_started":1}
I20260924 23:17:34.169145 12525 tablet_service.cc:1467] Processing CreateTablet for tablet 33d36b3812144fbd9c1a781d470b3ab9 (DEFAULT_TABLE table=test-table [id=0f4894e2f6c340999a088a17137d6262]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:17:34.169328 12394 tablet_service.cc:1467] Processing CreateTablet for tablet 33d36b3812144fbd9c1a781d470b3ab9 (DEFAULT_TABLE table=test-table [id=0f4894e2f6c340999a088a17137d6262]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:17:34.169538 12525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d36b3812144fbd9c1a781d470b3ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:17:34.169668 12394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d36b3812144fbd9c1a781d470b3ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:17:34.173098 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:34.162985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:40309 (local address 127.0.0.1:52148)
0924 23:17:34.163830 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 23:17:34.163839 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:17:34.163877 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:17:34.169435 (+ 5558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:17:34.169436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:17:34.169439 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:17:34.169492 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:17:34.169494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.170761 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.170762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:17:34.171827 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:17:34.171830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.172874 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.172875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:17:34.172953 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:34.172959 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 23:17:34.172988 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 23:17:34.173025 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":80,"threads_started":1}
I20260924 23:17:34.173471 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:34.163100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.1.4.195:40309)
0924 23:17:34.168762 (+ 5662us) server_negotiation.cc:206] Beginning negotiation
0924 23:17:34.168768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 23:17:34.168778 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:17:34.168816 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:17:34.168818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:17:34.168833 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:17:34.169569 (+ 736us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:17:34.169644 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.170663 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.172068 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.172789 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.173278 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:34.173361 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 23:17:34.173381 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 23:17:34.173414 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5621,"thread_start_us":78,"threads_started":1}
I20260924 23:17:34.173986 12677 tablet_service.cc:1467] Processing CreateTablet for tablet 33d36b3812144fbd9c1a781d470b3ab9 (DEFAULT_TABLE table=test-table [id=0f4894e2f6c340999a088a17137d6262]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:17:34.174252 12677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d36b3812144fbd9c1a781d470b3ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:17:34.175565 12763 tablet_bootstrap.cc:492] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279: Bootstrap starting.
I20260924 23:17:34.177066 12763 tablet_bootstrap.cc:654] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279: Neither blocks nor log segments found. Creating new log.
I20260924 23:17:34.177270 12762 tablet_bootstrap.cc:492] T 33d36b3812144fbd9c1a781d470b3ab9 P 3ad1a0bcadfb4063ad2ee5d527719062: Bootstrap starting.
I20260924 23:17:34.177292 12764 tablet_bootstrap.cc:492] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e: Bootstrap starting.
I20260924 23:17:34.178404 12764 tablet_bootstrap.cc:654] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e: Neither blocks nor log segments found. Creating new log.
I20260924 23:17:34.178418 12762 tablet_bootstrap.cc:654] T 33d36b3812144fbd9c1a781d470b3ab9 P 3ad1a0bcadfb4063ad2ee5d527719062: Neither blocks nor log segments found. Creating new log.
I20260924 23:17:34.178699 12764 log.cc:824] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e: Log is configured to *not* fsync() on all Append() calls
I20260924 23:17:34.178741 12763 log.cc:824] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279: Log is configured to *not* fsync() on all Append() calls
I20260924 23:17:34.179256 12762 log.cc:824] T 33d36b3812144fbd9c1a781d470b3ab9 P 3ad1a0bcadfb4063ad2ee5d527719062: Log is configured to *not* fsync() on all Append() calls
I20260924 23:17:34.179463 12764 tablet_bootstrap.cc:492] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e: No bootstrap required, opened a new log
I20260924 23:17:34.179531 12764 ts_tablet_manager.cc:1397] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:17:34.180948 12764 raft_consensus.cc:348] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fd68766c9c40888bae8b4911403279" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44565 } } peers { permanent_uuid: "95ad0ce3371a4c05a38e3199cc32326e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 40309 } } peers { permanent_uuid: "3ad1a0bcadfb4063ad2ee5d527719062" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 37063 } }
I20260924 23:17:34.181080 12764 raft_consensus.cc:374] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:17:34.181108 12764 raft_consensus.cc:729] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95ad0ce3371a4c05a38e3199cc32326e, State: Initialized, Role: FOLLOWER
I20260924 23:17:34.181222 12764 consensus_queue.cc:260] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fd68766c9c40888bae8b4911403279" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44565 } } peers { permanent_uuid: "95ad0ce3371a4c05a38e3199cc32326e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 40309 } } peers { permanent_uuid: "3ad1a0bcadfb4063ad2ee5d527719062" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 37063 } }
I20260924 23:17:34.181459 12764 ts_tablet_manager.cc:1428] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:17:34.181615 12743 heartbeater.cc:503] Master 127.1.4.254:39315 was elected leader, sending a full tablet report...
I20260924 23:17:34.183429 12763 tablet_bootstrap.cc:492] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279: No bootstrap required, opened a new log
I20260924 23:17:34.183498 12763 ts_tablet_manager.cc:1397] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260924 23:17:34.185527 12763 raft_consensus.cc:348] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fd68766c9c40888bae8b4911403279" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44565 } } peers { permanent_uuid: "95ad0ce3371a4c05a38e3199cc32326e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 40309 } } peers { permanent_uuid: "3ad1a0bcadfb4063ad2ee5d527719062" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 37063 } }
I20260924 23:17:34.185762 12763 raft_consensus.cc:374] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:17:34.185850 12763 raft_consensus.cc:729] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7fd68766c9c40888bae8b4911403279, State: Initialized, Role: FOLLOWER
I20260924 23:17:34.186250 12763 consensus_queue.cc:260] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fd68766c9c40888bae8b4911403279" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44565 } } peers { permanent_uuid: "95ad0ce3371a4c05a38e3199cc32326e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 40309 } } peers { permanent_uuid: "3ad1a0bcadfb4063ad2ee5d527719062" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 37063 } }
I20260924 23:17:34.186609 12763 ts_tablet_manager.cc:1428] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:17:34.186745 12609 heartbeater.cc:503] Master 127.1.4.254:39315 was elected leader, sending a full tablet report...
I20260924 23:17:34.187444 12762 tablet_bootstrap.cc:492] T 33d36b3812144fbd9c1a781d470b3ab9 P 3ad1a0bcadfb4063ad2ee5d527719062: No bootstrap required, opened a new log
I20260924 23:17:34.187518 12762 ts_tablet_manager.cc:1397] T 33d36b3812144fbd9c1a781d470b3ab9 P 3ad1a0bcadfb4063ad2ee5d527719062: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260924 23:17:34.189476 12762 raft_consensus.cc:348] T 33d36b3812144fbd9c1a781d470b3ab9 P 3ad1a0bcadfb4063ad2ee5d527719062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fd68766c9c40888bae8b4911403279" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44565 } } peers { permanent_uuid: "95ad0ce3371a4c05a38e3199cc32326e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 40309 } } peers { permanent_uuid: "3ad1a0bcadfb4063ad2ee5d527719062" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 37063 } }
I20260924 23:17:34.189667 12762 raft_consensus.cc:374] T 33d36b3812144fbd9c1a781d470b3ab9 P 3ad1a0bcadfb4063ad2ee5d527719062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:17:34.189715 12762 raft_consensus.cc:729] T 33d36b3812144fbd9c1a781d470b3ab9 P 3ad1a0bcadfb4063ad2ee5d527719062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad1a0bcadfb4063ad2ee5d527719062, State: Initialized, Role: FOLLOWER
I20260924 23:17:34.189862 12762 consensus_queue.cc:260] T 33d36b3812144fbd9c1a781d470b3ab9 P 3ad1a0bcadfb4063ad2ee5d527719062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fd68766c9c40888bae8b4911403279" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44565 } } peers { permanent_uuid: "95ad0ce3371a4c05a38e3199cc32326e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 40309 } } peers { permanent_uuid: "3ad1a0bcadfb4063ad2ee5d527719062" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 37063 } }
I20260924 23:17:34.190143 12471 heartbeater.cc:503] Master 127.1.4.254:39315 was elected leader, sending a full tablet report...
I20260924 23:17:34.190142 12762 ts_tablet_manager.cc:1428] T 33d36b3812144fbd9c1a781d470b3ab9 P 3ad1a0bcadfb4063ad2ee5d527719062: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
Sep 24 23:17:34 dist-test-slave-s1hw krb5kdc[11783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:17:34.275149 12768 raft_consensus.cc:482] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:17:34.275300 12768 raft_consensus.cc:504] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fd68766c9c40888bae8b4911403279" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44565 } } peers { permanent_uuid: "95ad0ce3371a4c05a38e3199cc32326e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 40309 } } peers { permanent_uuid: "3ad1a0bcadfb4063ad2ee5d527719062" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 37063 } }
I20260924 23:17:34.275913 12768 leader_election.cc:288] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7fd68766c9c40888bae8b4911403279 (127.1.4.194:44565), 3ad1a0bcadfb4063ad2ee5d527719062 (127.1.4.193:37063)
I20260924 23:17:34.279830 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:34.276155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:44565 (local address 127.1.4.195:51455)
0924 23:17:34.276420 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 23:17:34.276432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:17:34.276478 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:17:34.276901 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:17:34.276901 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:17:34.276903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:17:34.276972 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:17:34.276974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.277620 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.277621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:17:34.278806 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:17:34.278807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.279605 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.279606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:17:34.279674 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:34.279677 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:17:34.279717 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 23:17:34.279753 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260924 23:17:34.280102 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:34.276594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:51455 (local address 127.1.4.194:44565)
0924 23:17:34.276635 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:17:34.276638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:17:34.276646 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:17:34.276672 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:17:34.276673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:17:34.276683 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:17:34.276726 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:17:34.277041 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.277521 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.278910 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.279501 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.279564 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:34.279575 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:17:34.280005 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 23:17:34.280043 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:17:34.280404 12546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d36b3812144fbd9c1a781d470b3ab9" candidate_uuid: "95ad0ce3371a4c05a38e3199cc32326e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fd68766c9c40888bae8b4911403279" is_pre_election: true
I20260924 23:17:34.280586 12546 raft_consensus.cc:2454] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95ad0ce3371a4c05a38e3199cc32326e in term 0.
I20260924 23:17:34.280853 12630 leader_election.cc:302] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95ad0ce3371a4c05a38e3199cc32326e, f7fd68766c9c40888bae8b4911403279; no voters:
I20260924 23:17:34.280992 12768 raft_consensus.cc:2781] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:17:34.281034 12768 raft_consensus.cc:482] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:17:34.281056 12768 raft_consensus.cc:3037] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:17:34.281204 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:34.276592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:44239 (local address 127.1.4.193:37063)
0924 23:17:34.276638 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:17:34.276639 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:17:34.277699 (+ 1060us) server_negotiation.cc:407] Connection header received
0924 23:17:34.277785 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:17:34.277786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:17:34.277794 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:17:34.277835 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:17:34.278027 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.278720 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.280287 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.280812 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.280870 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:34.280880 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:17:34.281099 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 23:17:34.281146 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 23:17:34.281801 12771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:34.276238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:37063 (local address 127.1.4.195:44239)
0924 23:17:34.277645 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0924 23:17:34.277655 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:17:34.277721 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:17:34.277892 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:17:34.277893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:17:34.277894 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:17:34.277957 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:17:34.277958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.278804 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.278804 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:17:34.280197 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:17:34.280200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.280935 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.280935 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:17:34.281001 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:34.281005 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:17:34.281296 (+ 291us) client_negotiation.cc:240] Negotiation successful
0924 23:17:34.281714 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1390,"thread_start_us":100,"threads_started":1}
I20260924 23:17:34.281914 12768 raft_consensus.cc:504] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fd68766c9c40888bae8b4911403279" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44565 } } peers { permanent_uuid: "95ad0ce3371a4c05a38e3199cc32326e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 40309 } } peers { permanent_uuid: "3ad1a0bcadfb4063ad2ee5d527719062" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 37063 } }
I20260924 23:17:34.282028 12768 leader_election.cc:288] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [CANDIDATE]: Term 1 election: Requested vote from peers f7fd68766c9c40888bae8b4911403279 (127.1.4.194:44565), 3ad1a0bcadfb4063ad2ee5d527719062 (127.1.4.193:37063)
I20260924 23:17:34.282086 12425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d36b3812144fbd9c1a781d470b3ab9" candidate_uuid: "95ad0ce3371a4c05a38e3199cc32326e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad1a0bcadfb4063ad2ee5d527719062" is_pre_election: true
I20260924 23:17:34.282223 12425 raft_consensus.cc:2454] T 33d36b3812144fbd9c1a781d470b3ab9 P 3ad1a0bcadfb4063ad2ee5d527719062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95ad0ce3371a4c05a38e3199cc32326e in term 0.
I20260924 23:17:34.282518 12546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d36b3812144fbd9c1a781d470b3ab9" candidate_uuid: "95ad0ce3371a4c05a38e3199cc32326e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fd68766c9c40888bae8b4911403279"
I20260924 23:17:34.282573 12425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d36b3812144fbd9c1a781d470b3ab9" candidate_uuid: "95ad0ce3371a4c05a38e3199cc32326e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad1a0bcadfb4063ad2ee5d527719062"
I20260924 23:17:34.282598 12546 raft_consensus.cc:3037] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:17:34.282630 12425 raft_consensus.cc:3037] T 33d36b3812144fbd9c1a781d470b3ab9 P 3ad1a0bcadfb4063ad2ee5d527719062 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:17:34.283445 12425 raft_consensus.cc:2454] T 33d36b3812144fbd9c1a781d470b3ab9 P 3ad1a0bcadfb4063ad2ee5d527719062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95ad0ce3371a4c05a38e3199cc32326e in term 1.
I20260924 23:17:34.283440 12546 raft_consensus.cc:2454] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95ad0ce3371a4c05a38e3199cc32326e in term 1.
I20260924 23:17:34.283600 12630 leader_election.cc:302] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95ad0ce3371a4c05a38e3199cc32326e, f7fd68766c9c40888bae8b4911403279; no voters:
I20260924 23:17:34.283692 12768 raft_consensus.cc:2781] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:17:34.283839 12768 raft_consensus.cc:686] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [term 1 LEADER]: Becoming Leader. State: Replica: 95ad0ce3371a4c05a38e3199cc32326e, State: Running, Role: LEADER
I20260924 23:17:34.283932 12768 consensus_queue.cc:237] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fd68766c9c40888bae8b4911403279" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44565 } } peers { permanent_uuid: "95ad0ce3371a4c05a38e3199cc32326e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 40309 } } peers { permanent_uuid: "3ad1a0bcadfb4063ad2ee5d527719062" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 37063 } }
I20260924 23:17:34.284678 12259 catalog_manager.cc:5495] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e reported cstate change: term changed from 0 to 1, leader changed from <none> to 95ad0ce3371a4c05a38e3199cc32326e (127.1.4.195). New cstate: current_term: 1 leader_uuid: "95ad0ce3371a4c05a38e3199cc32326e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fd68766c9c40888bae8b4911403279" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95ad0ce3371a4c05a38e3199cc32326e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 40309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ad1a0bcadfb4063ad2ee5d527719062" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 37063 } health_report { overall_health: UNKNOWN } } }
I20260924 23:17:34.315907 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:34.312714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.1.4.195:40309)
0924 23:17:34.312758 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:17:34.312760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:17:34.312795 (+ 35us) server_negotiation.cc:407] Connection header received
0924 23:17:34.312881 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:17:34.312881 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:17:34.312903 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:17:34.312982 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:17:34.313115 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.313865 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.314946 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:17:34.315243 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:17:34.315311 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:17:34.315528 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:17:34.315667 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:17:34.315748 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 23:17:34.315794 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 23:17:34.315816 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:17:34.318081 12546 raft_consensus.cc:1261] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279 [term 1 FOLLOWER]: Refusing update from remote peer 95ad0ce3371a4c05a38e3199cc32326e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:17:34.318120 12425 raft_consensus.cc:1261] T 33d36b3812144fbd9c1a781d470b3ab9 P 3ad1a0bcadfb4063ad2ee5d527719062 [term 1 FOLLOWER]: Refusing update from remote peer 95ad0ce3371a4c05a38e3199cc32326e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:17:34.318534 12768 consensus_queue.cc:1035] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ad1a0bcadfb4063ad2ee5d527719062" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:17:34.318693 12772 consensus_queue.cc:1035] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7fd68766c9c40888bae8b4911403279" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:17:34.323073 12776 mvcc.cc:205] Tried to move back new op lower bound from 7333035435283906560 to 7333035435147538432. Current Snapshot: MvccSnapshot[applied={T|T < 7333035435283906560}]
I20260924 23:17:34.323397 12775 mvcc.cc:205] Tried to move back new op lower bound from 7333035435283906560 to 7333035435147538432. Current Snapshot: MvccSnapshot[applied={T|T < 7333035435283906560}]
I20260924 23:17:34.327221 12259 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:17:34.327323 12259 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:17:34.328181 12259 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3122b59aaf546c0a43f97f7462a52ee: Sending DeleteTablet for 3 replicas of tablet 33d36b3812144fbd9c1a781d470b3ab9
I20260924 23:17:34.328583 12525 tablet_service.cc:1514] Processing DeleteTablet for tablet 33d36b3812144fbd9c1a781d470b3ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:17:34 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:49220
I20260924 23:17:34.328761 12394 tablet_service.cc:1514] Processing DeleteTablet for tablet 33d36b3812144fbd9c1a781d470b3ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:17:34 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:48260
I20260924 23:17:34.329125 12677 tablet_service.cc:1514] Processing DeleteTablet for tablet 33d36b3812144fbd9c1a781d470b3ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:17:34 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:52148
I20260924 23:17:34.329763 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 12340
I20260924 23:17:34.333897 12784 tablet_replica.cc:330] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279: stopping tablet replica
I20260924 23:17:34.334100 12784 raft_consensus.cc:2229] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:17:34.334309 12784 raft_consensus.cc:2258] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:17:34.334857 12784 ts_tablet_manager.cc:1905] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:17:34.337258 12784 ts_tablet_manager.cc:1918] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:17:34.337404 12784 log.cc:1196] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/33d36b3812144fbd9c1a781d470b3ab9
I20260924 23:17:34.337724 12784 ts_tablet_manager.cc:1939] T 33d36b3812144fbd9c1a781d470b3ab9 P f7fd68766c9c40888bae8b4911403279: Deleting consensus metadata
I20260924 23:17:34.338230 12243 catalog_manager.cc:4841] TS f7fd68766c9c40888bae8b4911403279 (127.1.4.194:44565): tablet 33d36b3812144fbd9c1a781d470b3ab9 (table test-table [id=0f4894e2f6c340999a088a17137d6262]) successfully deleted
I20260924 23:17:34.339496 12786 tablet_replica.cc:330] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e: stopping tablet replica
I20260924 23:17:34.339674 12786 raft_consensus.cc:2229] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:17:34.339890 12786 raft_consensus.cc:2258] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:17:34.340453 12786 ts_tablet_manager.cc:1905] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:17:34.342199 12786 ts_tablet_manager.cc:1918] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:17:34.342319 12786 log.cc:1196] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/33d36b3812144fbd9c1a781d470b3ab9
I20260924 23:17:34.342628 12786 ts_tablet_manager.cc:1939] T 33d36b3812144fbd9c1a781d470b3ab9 P 95ad0ce3371a4c05a38e3199cc32326e: Deleting consensus metadata
I20260924 23:17:34.343091 12246 catalog_manager.cc:4841] TS 95ad0ce3371a4c05a38e3199cc32326e (127.1.4.195:40309): tablet 33d36b3812144fbd9c1a781d470b3ab9 (table test-table [id=0f4894e2f6c340999a088a17137d6262]) successfully deleted
W20260924 23:17:34.343117 12243 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.4.193:37063 (error 108)
I20260924 23:17:34.347817 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:17:34.347675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:37063 (local address 127.0.0.1:48270)
0924 23:17:34.347736 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 23:17:34.347773 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:37063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 23:17:34.348101 12243 catalog_manager.cc:4584] TS 3ad1a0bcadfb4063ad2ee5d527719062 (127.1.4.193:37063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33d36b3812144fbd9c1a781d470b3ab9: Network error: Client connection negotiation failed: client connection to 127.1.4.193:37063: connect: Connection refused (error 111)
I20260924 23:17:34.348187 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 12479
I20260924 23:17:34.354447 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 12614
I20260924 23:17:34.359906 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 12227
I20260924 23:17:34.365003 1043 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 23:17:34.699471 1043 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 23:17:34.699564 1043 mini_ranger.cc:66] Stopping Ranger...
I20260924 23:17:35.034264 1043 mini_ranger.cc:68] Stopped Ranger
2026-09-24 23:17:35.034 UTC [11813] LOG: received smart shutdown request
2026-09-24 23:17:35.038 UTC [11813] LOG: background worker "logical replication launcher" (PID 11822) exited with exit code 1
2026-09-24 23:17:35.038 UTC [11817] LOG: shutting down
2026-09-24 23:17:35.214 UTC [11813] LOG: database system is shut down
2026-09-24T23:17:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45905 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:17:35 dist-test-slave-s1hw krb5kdc[12796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:17:35 dist-test-slave-s1hw krb5kdc[12796](info): set up 2 sockets
Sep 24 23:17:35 dist-test-slave-s1hw krb5kdc[12796](info): commencing operation
krb5kdc: starting...
W20260924 23:17:36.255079 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:17:36 dist-test-slave-s1hw krb5kdc[12796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:17:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:17:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.4.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.4.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.4.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/rangeradmin_127.1.4.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.4.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/rangeradmin_127.1.4.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.4.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.4.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.4.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/rangerlookup_127.1.4.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.4.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/rangerlookup_127.1.4.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.4.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.4.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.4.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/HTTP_127.1.4.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.4.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/HTTP_127.1.4.212@KRBTEST.COM.keytab.
I20260924 23:17:36.417411 1043 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskEEvosl/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/postgres -l logfile start
2026-09-24 23:17:38.892 UTC [12826] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 23:17:38.892 UTC [12826] LOG: listening on IPv4 address "127.1.4.212", port 38509
2026-09-24 23:17:38.895 UTC [12826] LOG: listening on Unix socket "/tmp/.s.PGSQL.38509"
2026-09-24 23:17:38.922 UTC [12829] LOG: database system was shut down at 2026-09-24 23:17:37 UTC
2026-09-24 23:17:38.927 UTC [12826] LOG: database system is ready to accept connections
I20260924 23:17:39.790956 1043 mini_postgres.cc:96] Postgres bound to 38509
2026-09-24 23:17:39.796 UTC [12837] FATAL: database "slave" does not exist
127.1.4.212:38509 - accepting connections
I20260924 23:17:39.796970 1043 mini_ranger.cc:161] Starting Ranger...
I20260924 23:17:39.810590 1043 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 23:17:40.316813 1043 mini_ranger.cc:87] Created ranger Postgres database
I20260924 23:17:40.316884 1043 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-admin
2026-09-24 23:17:40,554 [I] DB FLAVOR :POSTGRES
2026-09-24 23:17:40,555 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 23:17:40,555 [I] Checking connection..
2026-09-24 23:17:40,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 23:17:40,790 [I] Checking connection passed.
2026-09-24 23:17:40,790 [I] --------- Verifying version history table ---------
2026-09-24 23:17:40,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 23:17:41,028 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 23:17:41,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 23:17:41,272 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 23:17:41,273 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 23:17:41,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 23:17:41.523 UTC [12901] WARNING: there is no transaction in progress
2026-09-24 23:17:41,529 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 23:17:41,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 23:17:41,762 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 23:17:41,763 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 23:17:41,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 23:17:42,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 23:17:42,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s1hw.c.gcp-upstream.internal','N') ;"
2026-09-24 23:17:42,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 23:17:42,732 [I] Table x_portal_user does not exist in database ranger
2026-09-24 23:17:42,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 23:17:42,969 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 23:17:42,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 23:17:43,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 23:17:43,472 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 23:17:43,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 23:17:44.458 UTC [13009] WARNING: there is no transaction in progress
2026-09-24 23:17:44.476 UTC [13009] WARNING: there is no transaction in progress
2026-09-24 23:17:44.495 UTC [13009] WARNING: there is no transaction in progress
2026-09-24 23:17:44.510 UTC [13009] WARNING: there is no transaction in progress
2026-09-24 23:17:44.528 UTC [13009] WARNING: there is no transaction in progress
2026-09-24 23:17:44.547 UTC [13009] WARNING: there is no transaction in progress
2026-09-24 23:17:44.605 UTC [13009] WARNING: there is no transaction in progress
2026-09-24 23:17:44.613 UTC [13009] WARNING: there is no transaction in progress
2026-09-24 23:17:44.628 UTC [13009] WARNING: there is no transaction in progress
2026-09-24 23:17:44.639 UTC [13009] WARNING: there is no transaction in progress
2026-09-24 23:17:44.649 UTC [13009] WARNING: there is no transaction in progress
2026-09-24 23:17:44.663 UTC [13009] WARNING: there is no transaction in progress
2026-09-24 23:17:44.673 UTC [13009] WARNING: there is no transaction in progress
2026-09-24 23:17:44.680 UTC [13009] WARNING: there is no transaction in progress
2026-09-24 23:17:44.688 UTC [13009] WARNING: there is no transaction in progress
2026-09-24 23:17:44.695 UTC [13009] WARNING: there is no transaction in progress
2026-09-24 23:17:45,384 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 23:17:45,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 23:17:45,623 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 23:17:45,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 23:17:45,884 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 23:17:45,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 23:17:46,120 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 23:17:46,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 23:17:46,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 23:17:46,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s1hw.c.gcp-upstream.internal';"
2026-09-24 23:17:46,849 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 23:17:46,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 23:17:47,095 [I] DB_PATCHES have already been applied
I20260924 23:17:47.100418 1043 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-admin:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskEEvosl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 23:17:47.100483 1043 mini_ranger.cc:193] Using host: 127.1.4.212
I20260924 23:17:47.103256 1043 mini_ranger.cc:227] Ranger admin URL: http://127.1.4.212:55615
Sep 24, 2026 11:17:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:17:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:17:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 11:17:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 11:17:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 11:17:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 11:17:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 23:17:48.077404 1043 mini_ranger.cc:160] Time spent starting Ranger: real 8.280s user 0.001s sys 0.007s
Sep 24, 2026 11:18:06 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 11:18:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 11:18:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 11:18:06 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 11:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 11:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 11:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 11:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 11:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 11:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 11:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 11:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 11:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 11:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 11:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 11:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 11:18:06 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 11:18:07 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 23:18:08.395390 1043 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.4.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.4.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.4.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/rangerkms_127.1.4.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.4.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/rangerkms_127.1.4.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.4.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.4.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.4.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/HTTP_127.1.4.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.4.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/HTTP_127.1.4.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 23:18:08 dist-test-slave-s1hw krb5kdc[12796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291888, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 23:18:08.497208 1043 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 23:18:08.506681 1043 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 23:18:09.432147 1043 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 23:18:09.432261 1043 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-kms
I20260924 23:18:09.432282 1043 mini_ranger_kms.cc:227] Using postgres at 127.1.4.212:38509
2026-09-24 23:18:09,597 [I] DB FLAVOR :POSTGRES
2026-09-24 23:18:09,598 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 23:18:09,598 [I] Checking connection
2026-09-24 23:18:09,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 23:18:09,850 [I] connection success
2026-09-24 23:18:09,850 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 23:18:09,850 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 23:18:09,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 23:18:10,121 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 23:18:10,121 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 23:18:10,121 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 23:18:10,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.4.212:38509/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 23:18:10,389 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 23:18:10.856796 1043 mini_ranger_kms.cc:310] Created kms service
I20260924 23:18:11.070341 1043 mini_ranger_kms.cc:326] Created kudu user
I20260924 23:18:11.110431 1043 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 23:18:11 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291888, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:18:11.496544 1043 mini_ranger_kms.cc:384] Added ranger policy
I20260924 23:18:11.496649 1043 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-kms:/tmp/dist-test-taskEEvosl/build/release/bin/postgresql.jar:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskEEvosl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 23:18:11.496667 1043 mini_ranger_kms.cc:240] Using host: 127.1.4.212
I20260924 23:18:11.499898 1043 mini_ranger_kms.cc:276] Ranger KMS PID: 13189
I20260924 23:18:11.499959 1043 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.4.212:34691
Sep 24, 2026 11:18:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:18:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:18:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 11:18:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 11:18:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 11:18:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 23:18:13.562384 1043 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.065s user 0.009s sys 0.135s
I20260924 23:18:13.562489 1043 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 23:18:13.562536 1043 mini_ranger_kms.cc:401] 127.1.4.212:34691/kms/v1/keys
Sep 24 23:18:13 dist-test-slave-s1hw krb5kdc[12796](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790291893, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.4.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 23:18:14 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291893, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.4.212@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
Sep 24 23:18:14 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291893, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.4.212@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
Sep 24 23:18:16 dist-test-slave-s1hw krb5kdc[12796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:18:16.847626 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:46061
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:46061
--ranger_config_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.4.212:34691/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:16.944041 13240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:16.944252 13240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:16.944272 13240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:16.945493 13240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:18:16.945539 13240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:16.945554 13240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:16.945564 13240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:18:16.945574 13240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:18:16.947131 13240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.4.212:34691/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:46061
--ranger_config_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:46061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:16.947352 13240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:16.947594 13240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:16.952256 13248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:16.952404 13240 server_base.cc:1034] running on GCE node
W20260924 23:18:16.952468 13246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:16.952487 13245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:16.952847 13240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:16.953126 13240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:16.954309 13240 hybrid_clock.cc:648] HybridClock initialized: now 1790291896954293 us; error 29 us; skew 500 ppm
Sep 24 23:18:16 dist-test-slave-s1hw krb5kdc[12796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291896, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:16.960575 13240 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:18:16.961056 13240 webserver.cc:460] Webserver started at http://127.1.4.254:42563/ using document root <none> and password file <none>
I20260924 23:18:16.961272 13240 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:16.961318 13240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:16.961439 13240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 23:18:16 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291896, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:18:17.021507 13240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "1c09dceb26a9475984319cdeee4d2f93"
format_stamp: "Formatted at 2026-09-24 23:18:17 on dist-test-slave-s1hw"
server_key: "b455a22ddbd692cb87086c3b1833b975"
server_key_iv: "97070e42e15ea4b8f772255f550cacda"
server_key_version: "kuduclusterkey@0"
I20260924 23:18:17.021860 13240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "1c09dceb26a9475984319cdeee4d2f93"
format_stamp: "Formatted at 2026-09-24 23:18:17 on dist-test-slave-s1hw"
server_key: "b455a22ddbd692cb87086c3b1833b975"
server_key_iv: "97070e42e15ea4b8f772255f550cacda"
server_key_version: "kuduclusterkey@0"
I20260924 23:18:17.023227 13240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:18:17.035189 13255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:17.035482 13240 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:17.035586 13240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "1c09dceb26a9475984319cdeee4d2f93"
format_stamp: "Formatted at 2026-09-24 23:18:17 on dist-test-slave-s1hw"
server_key: "b455a22ddbd692cb87086c3b1833b975"
server_key_iv: "97070e42e15ea4b8f772255f550cacda"
server_key_version: "kuduclusterkey@0"
I20260924 23:18:17.035684 13240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:17.056823 13240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:17.058730 13240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:17.058874 13240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:17.063118 13240 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:46061
I20260924 23:18:17.063186 13307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:46061 every 8 connection(s)
I20260924 23:18:17.063545 13240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:18:17.064213 13308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:17.064254 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 13240
I20260924 23:18:17.064327 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:18:17.064514 1043 external_mini_cluster.cc:1428] Setting key 9e7f8807f1fcb8e1ad2246113219935f
I20260924 23:18:17.067569 13308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93: Bootstrap starting.
I20260924 23:18:17.068140 13308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:17.068395 13308 log.cc:824] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:17.069173 13308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93: No bootstrap required, opened a new log
I20260924 23:18:17.071400 13308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09dceb26a9475984319cdeee4d2f93" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46061 } }
I20260924 23:18:17.071578 13308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:17.071611 13308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c09dceb26a9475984319cdeee4d2f93, State: Initialized, Role: FOLLOWER
I20260924 23:18:17.071802 13308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09dceb26a9475984319cdeee4d2f93" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46061 } }
I20260924 23:18:17.071884 13308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:18:17.071916 13308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:18:17.071975 13308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 23:18:17 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:17.072628 13308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09dceb26a9475984319cdeee4d2f93" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46061 } }
I20260924 23:18:17.072727 13308 leader_election.cc:302] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c09dceb26a9475984319cdeee4d2f93; no voters:
I20260924 23:18:17.072894 13308 leader_election.cc:288] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:18:17.073172 13308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:18:17.074131 13314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:18:17.074318 13314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 1 LEADER]: Becoming Leader. State: Replica: 1c09dceb26a9475984319cdeee4d2f93, State: Running, Role: LEADER
I20260924 23:18:17.074455 13314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09dceb26a9475984319cdeee4d2f93" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46061 } }
I20260924 23:18:17.074834 13320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c09dceb26a9475984319cdeee4d2f93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09dceb26a9475984319cdeee4d2f93" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46061 } } }
I20260924 23:18:17.074983 13320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [sys.catalog]: This master's current role is: LEADER
I20260924 23:18:17.075091 13308 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 23:18:17.075299 13320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c09dceb26a9475984319cdeee4d2f93. Latest consensus state: current_term: 1 leader_uuid: "1c09dceb26a9475984319cdeee4d2f93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09dceb26a9475984319cdeee4d2f93" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46061 } } }
I20260924 23:18:17.075449 13320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [sys.catalog]: This master's current role is: LEADER
I20260924 23:18:17.078496 13327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:18:17.079285 13311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:17.065028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.1.4.254:46061)
0924 23:18:17.065198 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:17.065208 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:17.065245 (+ 37us) server_negotiation.cc:407] Connection header received
0924 23:18:17.065954 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:17.065964 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:17.066030 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:17.066171 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:17.066923 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:17.068385 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:17.069398 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:17.070040 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:17.070151 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:17.072894 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:17.072898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:17.072898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:17.072910 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:17.074832 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:17.075244 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:17.075247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:17.075248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:17.075343 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:17.075590 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:17.075591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:17.075591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:17.077449 (+ 1858us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:17.077877 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:17.078483 (+ 606us) server_negotiation.cc:299] Negotiation successful
0924 23:18:17.078963 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260924 23:18:17.079494 13327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:18:17.086818 13327 catalog_manager.cc:1347] Generated new cluster ID: 17e5b632641f45e6b4e44b81b8f57b34
I20260924 23:18:17.086871 13327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:18:17.107264 13327 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:18:17.108217 13327 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:18:17.119437 13327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93: Generated new TSK 0
I20260924 23:18:17.119791 13327 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:18:17.846922 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.4.212:34691/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:46061
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:17.932600 13353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:17.932808 13353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:17.932839 13353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:17.934000 13353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:17.934041 13353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:17.934074 13353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:18:17.935563 13353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.4.212:34691/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:46061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:17.935794 13353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:17.936008 13353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:17.941156 13358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:17.941174 13361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:17.941473 13359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:17.941565 13353 server_base.cc:1034] running on GCE node
I20260924 23:18:17.941809 13353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:17.942059 13353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:17.943210 13353 hybrid_clock.cc:648] HybridClock initialized: now 1790291897943198 us; error 31 us; skew 500 ppm
Sep 24 23:18:17 dist-test-slave-s1hw krb5kdc[12796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291897, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:17.946091 13353 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:18:17.946642 13353 webserver.cc:460] Webserver started at http://127.1.4.193:39417/ using document root <none> and password file <none>
I20260924 23:18:17.946904 13353 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:17.946973 13353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:17.947108 13353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 23:18:17 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291897, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:18:17.961997 13353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "12bfcf8b23734d428c6709a4c4d8599f"
format_stamp: "Formatted at 2026-09-24 23:18:17 on dist-test-slave-s1hw"
server_key: "f19aebaed7e780c2ef765d9391d35435"
server_key_iv: "6c29a674a870883e42130d20a73e38d6"
server_key_version: "kuduclusterkey@0"
I20260924 23:18:17.962504 13353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "12bfcf8b23734d428c6709a4c4d8599f"
format_stamp: "Formatted at 2026-09-24 23:18:17 on dist-test-slave-s1hw"
server_key: "f19aebaed7e780c2ef765d9391d35435"
server_key_iv: "6c29a674a870883e42130d20a73e38d6"
server_key_version: "kuduclusterkey@0"
I20260924 23:18:17.967695 13353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260924 23:18:17.990834 13368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:17.991046 13353 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260924 23:18:17.991130 13353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "12bfcf8b23734d428c6709a4c4d8599f"
format_stamp: "Formatted at 2026-09-24 23:18:17 on dist-test-slave-s1hw"
server_key: "f19aebaed7e780c2ef765d9391d35435"
server_key_iv: "6c29a674a870883e42130d20a73e38d6"
server_key_version: "kuduclusterkey@0"
I20260924 23:18:17.991226 13353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:18.029418 13353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:18.032604 13353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:18.032773 13353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:18.033025 13353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:18.033367 13353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:18:18.033411 13353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:18.033439 13353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:18:18.033459 13353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:18.041129 13353 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:44999
I20260924 23:18:18.041400 13483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:44999 every 8 connection(s)
I20260924 23:18:18.041555 13353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:18:18.043459 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 13353
I20260924 23:18:18.043555 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:18:18.043701 1043 external_mini_cluster.cc:1428] Setting key dbb0c184fdcdaae8c55c77b9bbf97e1f
Sep 24 23:18:18 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291897, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:18.053417 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:18.042322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:46061 (local address 127.1.4.193:53629)
0924 23:18:18.042500 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:18.042552 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:18.043335 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:18.043853 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:18.043858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:18.044107 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:18.044416 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:18.044423 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:18.045416 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:18.045417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:18.046674 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:18.046677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:18.047203 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:18.047204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:18.047359 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:18.047970 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:18.047981 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:18.049584 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:18.051687 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:18.051693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:18.051696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:18.052018 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:18.052257 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:18.052258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:18.052258 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:18.052404 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:18.052702 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:18.052722 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:18.052909 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 23:18:18.053025 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 23:18:18.053181 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260924 23:18:18.053890 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:18.042335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:53629 (local address 127.1.4.254:46061)
0924 23:18:18.042613 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:18.042615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:18.042627 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:18:18.043547 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:18.043549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:18.043563 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:18.043629 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:18.044509 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:18.045315 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:18.046777 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:18.047098 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:18.047166 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:18.049691 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:18.049692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:18.049693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:18.049700 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:18.051585 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:18.052105 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:18.052106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:18.052107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:18.052175 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:18.052479 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:18.052480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:18.052480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:18.052617 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:18.052676 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:18.053687 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0924 23:18:18.053782 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":73,"threads_started":1}
I20260924 23:18:18.054459 13484 heartbeater.cc:347] Connected to a master server at 127.1.4.254:46061
I20260924 23:18:18.054557 13484 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:18.054767 13484 heartbeater.cc:511] Master 127.1.4.254:46061 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
I20260924 23:18:18.059868 13272 ts_manager.cc:194] Registered new tserver with Master: 12bfcf8b23734d428c6709a4c4d8599f (127.1.4.193:44999)
I20260924 23:18:18.060753 13272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:53629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:18:18.078727 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.4.212:34691/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:46061
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:18.181034 13490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:18.181242 13490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:18.181273 13490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:18.182457 13490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:18.182503 13490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:18.182538 13490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:18:18.184196 13490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.4.212:34691/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:46061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:18.184428 13490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:18.184679 13490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:18.189868 13497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:18.189882 13498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:18.189870 13500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:18.190356 13490 server_base.cc:1034] running on GCE node
I20260924 23:18:18.190585 13490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:18.190841 13490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:18.191991 13490 hybrid_clock.cc:648] HybridClock initialized: now 1790291898191977 us; error 27 us; skew 500 ppm
Sep 24 23:18:18 dist-test-slave-s1hw krb5kdc[12796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291898, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:18.194577 13490 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:18:18.195055 13490 webserver.cc:460] Webserver started at http://127.1.4.194:41125/ using document root <none> and password file <none>
I20260924 23:18:18.195297 13490 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:18.195346 13490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:18.195536 13490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 23:18:18 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291898, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:18:18.207846 13490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "bf887ee655014b588e56493ea326d169"
format_stamp: "Formatted at 2026-09-24 23:18:18 on dist-test-slave-s1hw"
server_key: "309c8e31d21f8815bdd60e1f87808550"
server_key_iv: "582a616adda3fffcb1a61a96be751021"
server_key_version: "kuduclusterkey@0"
I20260924 23:18:18.208343 13490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "bf887ee655014b588e56493ea326d169"
format_stamp: "Formatted at 2026-09-24 23:18:18 on dist-test-slave-s1hw"
server_key: "309c8e31d21f8815bdd60e1f87808550"
server_key_iv: "582a616adda3fffcb1a61a96be751021"
server_key_version: "kuduclusterkey@0"
I20260924 23:18:18.211191 13490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
Sep 24 23:18:18 dist-test-slave-s1hw krb5kdc[12796](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790291898, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:18.228654 13507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:18.228883 13490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 23:18:18.228971 13490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "bf887ee655014b588e56493ea326d169"
format_stamp: "Formatted at 2026-09-24 23:18:18 on dist-test-slave-s1hw"
server_key: "309c8e31d21f8815bdd60e1f87808550"
server_key_iv: "582a616adda3fffcb1a61a96be751021"
server_key_version: "kuduclusterkey@0"
I20260924 23:18:18.229070 13490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:18.247133 13490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:18.249434 13490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:18.249590 13490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:18.249868 13490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:18.250270 13490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:18:18.250322 13490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:18.250357 13490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:18:18.250375 13490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:18.256707 13490 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:32947
I20260924 23:18:18.257153 13490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:18:18.264680 13620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:32947 every 8 connection(s)
I20260924 23:18:18.266822 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 13490
I20260924 23:18:18.266933 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:18:18.267095 1043 external_mini_cluster.cc:1428] Setting key 1ab6a41bf835a23f97fc2435adaaaf7a
Sep 24 23:18:18 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291898, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:18.275635 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:18.265408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:50235 (local address 127.1.4.254:46061)
0924 23:18:18.265459 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:18.265461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:18.265721 (+ 260us) server_negotiation.cc:407] Connection header received
0924 23:18:18.266209 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:18.266210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:18.266223 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:18.266276 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:18.266824 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:18.267528 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:18.269532 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:18.269817 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:18.269878 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:18.272357 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:18.272359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:18.272359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:18.272367 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:18.273991 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:18.274431 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:18.274432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:18.274433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:18.274492 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:18.274739 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:18.274740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:18.274740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:18.274873 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:18.274935 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:18.275450 (+ 515us) server_negotiation.cc:299] Negotiation successful
0924 23:18:18.275537 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
I20260924 23:18:18.276273 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:18.265383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:46061 (local address 127.1.4.194:50235)
0924 23:18:18.265565 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:18.265616 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:18.266113 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:18.266346 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:18.266350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:18.266542 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:18.266731 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:18.266735 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:18.267604 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:18.267605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:18.269445 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:18.269448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:18.269933 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:18.269935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:18.270083 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:18.270653 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:18.270662 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:18.272252 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:18.274066 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:18.274071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:18.274074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:18.274351 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:18.274556 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:18.274556 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:18.274557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:18.274677 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:18.274967 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:18.274974 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:18.275161 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 23:18:18.275275 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 23:18:18.276059 (+ 784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260924 23:18:18.276703 13621 heartbeater.cc:347] Connected to a master server at 127.1.4.254:46061
I20260924 23:18:18.276793 13621 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:18.276973 13621 heartbeater.cc:511] Master 127.1.4.254:46061 requested a full tablet report, sending...
I20260924 23:18:18.277350 13272 ts_manager.cc:194] Registered new tserver with Master: bf887ee655014b588e56493ea326d169 (127.1.4.194:32947)
I20260924 23:18:18.277705 13272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:50235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:18:18.295485 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.4.212:34691/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:46061
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:18.391938 13627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:18.392144 13627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:18.392176 13627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:18.393380 13627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:18.393421 13627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:18.393456 13627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:18:18.394982 13627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.4.212:34691/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:46061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:18.395324 13627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:18.395627 13627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:18.400528 13633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:18.400511 13635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:18.400491 13632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:18.400830 13627 server_base.cc:1034] running on GCE node
I20260924 23:18:18.401069 13627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:18.401302 13627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:18.402460 13627 hybrid_clock.cc:648] HybridClock initialized: now 1790291898402447 us; error 24 us; skew 500 ppm
Sep 24 23:18:18 dist-test-slave-s1hw krb5kdc[12796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291898, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:18.405001 13627 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:18:18.405445 13627 webserver.cc:460] Webserver started at http://127.1.4.195:39791/ using document root <none> and password file <none>
I20260924 23:18:18.405670 13627 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:18.405717 13627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:18.405860 13627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 23:18:18 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291898, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:18:18.416508 13627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "81726cb75cf24e5693d166e92dd308b9"
format_stamp: "Formatted at 2026-09-24 23:18:18 on dist-test-slave-s1hw"
server_key: "f382e8683063cade74a899e08eeb44f5"
server_key_iv: "c9db5970fc05d84fcd02a31a7e8796c7"
server_key_version: "kuduclusterkey@0"
I20260924 23:18:18.416923 13627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "81726cb75cf24e5693d166e92dd308b9"
format_stamp: "Formatted at 2026-09-24 23:18:18 on dist-test-slave-s1hw"
server_key: "f382e8683063cade74a899e08eeb44f5"
server_key_iv: "c9db5970fc05d84fcd02a31a7e8796c7"
server_key_version: "kuduclusterkey@0"
I20260924 23:18:18.420253 13627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 23:18:18.443073 13642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:18.443403 13627 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260924 23:18:18.443519 13627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "81726cb75cf24e5693d166e92dd308b9"
format_stamp: "Formatted at 2026-09-24 23:18:18 on dist-test-slave-s1hw"
server_key: "f382e8683063cade74a899e08eeb44f5"
server_key_iv: "c9db5970fc05d84fcd02a31a7e8796c7"
server_key_version: "kuduclusterkey@0"
I20260924 23:18:18.443620 13627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:18.453609 13627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:18.456159 13627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:18.456317 13627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:18.456569 13627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:18.456964 13627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:18:18.457011 13627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:18.457041 13627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:18:18.457056 13627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:18.463727 13627 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:36019
I20260924 23:18:18.463769 13755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:36019 every 8 connection(s)
I20260924 23:18:18.464169 13627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
Sep 24 23:18:18 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291898, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:18.474339 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 13627
I20260924 23:18:18.474434 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:18:18.474601 1043 external_mini_cluster.cc:1428] Setting key d9a8c2421a49e0f45e82b3caa4c16edf
I20260924 23:18:18.475179 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:18.465301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:52819 (local address 127.1.4.254:46061)
0924 23:18:18.465343 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:18.465345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:18.465355 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:18:18.465846 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:18.465847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:18.465858 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:18.465906 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:18.466465 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:18.466960 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:18.468426 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:18.468733 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:18.468788 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:18.471382 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:18.471384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:18.471384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:18.471393 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:18.473142 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:18.473745 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:18.473747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:18.473748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:18.473811 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:18.474122 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:18.474123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:18.474123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:18.474264 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:18.474307 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:18.474990 (+ 683us) server_negotiation.cc:299] Negotiation successful
0924 23:18:18.475080 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:18:18.475876 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:18.465061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:46061 (local address 127.1.4.195:52819)
0924 23:18:18.465221 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:18.465264 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:18.465744 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:18.465991 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:18.465996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:18.466178 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:18.466381 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:18.466385 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:18.467045 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:18.467046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:18.468319 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:18.468323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:18.468893 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:18.468894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:18.469040 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:18.469642 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:18.469652 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:18.471243 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:18.473327 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:18.473335 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:18.473338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:18.473651 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:18.473895 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:18.473896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:18.473896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:18.474042 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:18.474428 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:18.474435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:18.474622 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 23:18:18.475459 (+ 837us) client_negotiation.cc:240] Negotiation successful
0924 23:18:18.475635 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260924 23:18:18.476336 13756 heartbeater.cc:347] Connected to a master server at 127.1.4.254:46061
I20260924 23:18:18.476447 13756 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:18.476641 13756 heartbeater.cc:511] Master 127.1.4.254:46061 requested a full tablet report, sending...
I20260924 23:18:18.477083 13272 ts_manager.cc:194] Registered new tserver with Master: 81726cb75cf24e5693d166e92dd308b9 (127.1.4.195:36019)
I20260924 23:18:18.477581 13272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:52819
I20260924 23:18:18.486431 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:18:18.487015 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 13353
I20260924 23:18:18.491737 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 13490
I20260924 23:18:18.496275 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 13627
I20260924 23:18:18.500097 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 13240
I20260924 23:18:18.504312 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:46061
--webserver_interface=127.1.4.254
--webserver_port=42563
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:46061
--ranger_config_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.4.212:34691/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.4.212:34691/kms with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:18.584097 13759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:18.584312 13759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:18.584334 13759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:18.585454 13759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:18:18.585492 13759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:18.585505 13759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:18.585515 13759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:18:18.585526 13759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:18:18.587029 13759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.4.212:34691/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:46061
--ranger_config_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:46061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=42563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:18.587219 13759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:18.587416 13759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:18.592123 13765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:18.592149 13767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:18.592147 13764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:18.592350 13759 server_base.cc:1034] running on GCE node
I20260924 23:18:18.592581 13759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:18.592880 13759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:18.594045 13759 hybrid_clock.cc:648] HybridClock initialized: now 1790291898594028 us; error 32 us; skew 500 ppm
Sep 24 23:18:18 dist-test-slave-s1hw krb5kdc[12796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291898, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:18.596849 13759 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:18:18.597292 13759 webserver.cc:460] Webserver started at http://127.1.4.254:42563/ using document root <none> and password file <none>
I20260924 23:18:18.597496 13759 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:18.597546 13759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 23:18:18 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291898, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:18:18.639313 13759 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:18.640093 13775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:18.640300 13759 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:18.640398 13759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "1c09dceb26a9475984319cdeee4d2f93"
format_stamp: "Formatted at 2026-09-24 23:18:17 on dist-test-slave-s1hw"
server_key: "b455a22ddbd692cb87086c3b1833b975"
server_key_iv: "97070e42e15ea4b8f772255f550cacda"
server_key_version: "kuduclusterkey@0"
I20260924 23:18:18.640902 13759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:18.667788 13759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:18.670050 13759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:18.670197 13759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:18.675067 13759 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:46061
I20260924 23:18:18.675098 13827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:46061 every 8 connection(s)
I20260924 23:18:18.675575 13759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:18:18.676885 13828 sys_catalog.cc:263] Verifying existing consensus state
I20260924 23:18:18.677518 13828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93: Bootstrap starting.
I20260924 23:18:18.679594 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 13759
I20260924 23:18:18.679890 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.4.212:34691/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:44999
--local_ip_for_outbound_sockets=127.1.4.193
--tserver_master_addrs=127.1.4.254:46061
--webserver_port=39417
--webserver_interface=127.1.4.193
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.4.212:34691/kms with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:18:18.685892 13828 log.cc:824] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:18.687975 13828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:18:18.688251 13828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93: Bootstrap complete.
I20260924 23:18:18.690464 13828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09dceb26a9475984319cdeee4d2f93" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46061 } }
I20260924 23:18:18.690701 13828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c09dceb26a9475984319cdeee4d2f93, State: Initialized, Role: FOLLOWER
I20260924 23:18:18.690862 13828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09dceb26a9475984319cdeee4d2f93" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46061 } }
I20260924 23:18:18.690927 13828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:18:18.690968 13828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:18:18.691012 13828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 1 FOLLOWER]: Advancing to term 2
I20260924 23:18:18.692086 13828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09dceb26a9475984319cdeee4d2f93" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46061 } }
I20260924 23:18:18.692178 13828 leader_election.cc:302] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c09dceb26a9475984319cdeee4d2f93; no voters:
I20260924 23:18:18.692348 13828 leader_election.cc:288] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 23:18:18.692394 13832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 23:18:18.692507 13832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [term 2 LEADER]: Becoming Leader. State: Replica: 1c09dceb26a9475984319cdeee4d2f93, State: Running, Role: LEADER
I20260924 23:18:18.692615 13828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:18:18.692596 13832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09dceb26a9475984319cdeee4d2f93" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46061 } }
I20260924 23:18:18.692956 13834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1c09dceb26a9475984319cdeee4d2f93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09dceb26a9475984319cdeee4d2f93" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46061 } } }
I20260924 23:18:18.692966 13833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c09dceb26a9475984319cdeee4d2f93. Latest consensus state: current_term: 2 leader_uuid: "1c09dceb26a9475984319cdeee4d2f93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c09dceb26a9475984319cdeee4d2f93" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 46061 } } }
I20260924 23:18:18.693123 13834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [sys.catalog]: This master's current role is: LEADER
I20260924 23:18:18.693135 13833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93 [sys.catalog]: This master's current role is: LEADER
I20260924 23:18:18.693462 13841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:18:18.693969 13841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:18:18.694105 13841 catalog_manager.cc:1259] Loaded cluster ID: 17e5b632641f45e6b4e44b81b8f57b34
I20260924 23:18:18.694145 13841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:18:18.694435 13828 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 23:18:18.697491 13841 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:18:18.698021 13841 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93: Loaded TSK: 0
I20260924 23:18:18.698318 13841 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 23:18:18.764864 13830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:18.765064 13830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:18.765094 13830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:18.766594 13830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:18.766649 13830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:18.766685 13830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:18:18.768834 13830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.4.212:34691/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:44999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=39417
--tserver_master_addrs=127.1.4.254:46061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:18.769083 13830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:18.769349 13830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:18.775539 13866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:18.775548 13865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:18.775887 13868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:18.776503 13830 server_base.cc:1034] running on GCE node
I20260924 23:18:18.776767 13830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:18.777051 13830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:18.778235 13830 hybrid_clock.cc:648] HybridClock initialized: now 1790291898778200 us; error 40 us; skew 500 ppm
Sep 24 23:18:18 dist-test-slave-s1hw krb5kdc[12796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291898, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:18.781406 13830 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:18:18.781970 13830 webserver.cc:460] Webserver started at http://127.1.4.193:39417/ using document root <none> and password file <none>
I20260924 23:18:18.782243 13830 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:18.782320 13830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 23:18:18 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291898, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:18:18.804018 13830 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260924 23:18:18.805042 13876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:18.805194 13830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:18.805287 13830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "12bfcf8b23734d428c6709a4c4d8599f"
format_stamp: "Formatted at 2026-09-24 23:18:17 on dist-test-slave-s1hw"
server_key: "f19aebaed7e780c2ef765d9391d35435"
server_key_iv: "6c29a674a870883e42130d20a73e38d6"
server_key_version: "kuduclusterkey@0"
I20260924 23:18:18.805773 13830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:18.818833 13830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:18.821337 13830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:18.821506 13830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:18.821754 13830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:18.822125 13830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:18:18.822175 13830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:18.822206 13830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:18:18.822243 13830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:18.830268 13830 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:44999
I20260924 23:18:18.830785 13830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:18:18.832572 13989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:44999 every 8 connection(s)
I20260924 23:18:18.835482 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 13830
I20260924 23:18:18.835743 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.4.212:34691/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:32947
--local_ip_for_outbound_sockets=127.1.4.194
--tserver_master_addrs=127.1.4.254:46061
--webserver_port=41125
--webserver_interface=127.1.4.194
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.4.212:34691/kms with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 23:18:18 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291898, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:18.854416 13992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:18.832301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:46061 (local address 127.1.4.193:52099)
0924 23:18:18.832838 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:18.832884 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:18.833379 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:18.833928 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:18.833931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:18.834103 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:18.834301 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:18.834305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:18.835858 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:18.835860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:18.837098 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:18.837102 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:18.838011 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:18.838012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:18.838142 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:18.838768 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:18.838778 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:18.841313 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:18.848198 (+ 6885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:18.848209 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:18.848212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:18.852178 (+ 3966us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:18.852933 (+ 755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:18.852934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:18.852934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:18.853086 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:18.853543 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:18.853549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:18.853854 (+ 305us) client_negotiation.cc:769] Sending connection context
0924 23:18:18.853976 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 23:18:18.854153 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":77,"threads_started":1}
I20260924 23:18:18.857437 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:18.832680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:52099 (local address 127.1.4.254:46061)
0924 23:18:18.832895 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:18.832902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:18.832977 (+ 75us) server_negotiation.cc:407] Connection header received
0924 23:18:18.833603 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:18.833609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:18.833653 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:18.833766 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:18.834409 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:18.835736 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:18.837192 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:18.837806 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:18.837901 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:18.841464 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:18.841469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:18.841470 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:18.841483 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:18.848076 (+ 6593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:18.852685 (+ 4609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:18.852689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:18.852689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:18.852858 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:18.853147 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:18.853148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:18.853148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:18.853447 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:18.853517 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:18.857040 (+ 3523us) server_negotiation.cc:299] Negotiation successful
0924 23:18:18.857191 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":85,"threads_started":1}
I20260924 23:18:18.859953 13990 heartbeater.cc:347] Connected to a master server at 127.1.4.254:46061
I20260924 23:18:18.860055 13990 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:18.860244 13990 heartbeater.cc:511] Master 127.1.4.254:46061 requested a full tablet report, sending...
W20260924 23:18:18.861146 13990 heartbeater.cc:644] Failed to heartbeat to 127.1.4.254:46061 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 23:18:18.861214 13990 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:18.861287 13990 heartbeater.cc:511] Master 127.1.4.254:46061 requested a full tablet report, sending...
I20260924 23:18:18.861666 13990 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:18.861745 13990 heartbeater.cc:511] Master 127.1.4.254:46061 requested a full tablet report, sending...
W20260924 23:18:18.862054 13990 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 23:18:18.947714 13994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:18.948084 13994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:18.948171 13994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:18.950008 13994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:18.950073 13994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:18.950114 13994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:18:18.952450 13994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.4.212:34691/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:32947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=41125
--tserver_master_addrs=127.1.4.254:46061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:18.952702 13994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:18.952976 13994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:18.958246 14000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:18.958261 14002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:18.958343 13994 server_base.cc:1034] running on GCE node
W20260924 23:18:18.958235 13999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:18.958648 13994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:18.958897 13994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:18.960040 13994 hybrid_clock.cc:648] HybridClock initialized: now 1790291898960023 us; error 32 us; skew 500 ppm
Sep 24 23:18:18 dist-test-slave-s1hw krb5kdc[12796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291898, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:18.962496 13994 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:18:18.962942 13994 webserver.cc:460] Webserver started at http://127.1.4.194:41125/ using document root <none> and password file <none>
I20260924 23:18:18.963158 13994 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:18.963219 13994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 23:18:18 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291898, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:18:18.983683 13994 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:18.984740 14010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:18.984942 13994 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:18.985028 13994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "bf887ee655014b588e56493ea326d169"
format_stamp: "Formatted at 2026-09-24 23:18:18 on dist-test-slave-s1hw"
server_key: "309c8e31d21f8815bdd60e1f87808550"
server_key_iv: "582a616adda3fffcb1a61a96be751021"
server_key_version: "kuduclusterkey@0"
I20260924 23:18:18.985492 13994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:19.000241 13994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:19.002151 13994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:19.002290 13994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:19.002498 13994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:19.002826 13994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:18:19.002874 13994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:19.002936 13994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:18:19.002962 13994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:19.009446 13994 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:32947
I20260924 23:18:19.009510 14123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:32947 every 8 connection(s)
I20260924 23:18:19.009897 13994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:18:19.013536 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 13994
I20260924 23:18:19.013792 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.4.212:34691/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:36019
--local_ip_for_outbound_sockets=127.1.4.195
--tserver_master_addrs=127.1.4.254:46061
--webserver_port=39791
--webserver_interface=127.1.4.195
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.4.212:34691/kms with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 23:18:19 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291898, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:19.023463 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:19.010938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:44661 (local address 127.1.4.254:46061)
0924 23:18:19.010981 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:19.010983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:19.010992 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:18:19.011606 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:19.011607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:19.011624 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:19.011654 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:19.012493 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:19.013297 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:19.016835 (+ 3538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:19.017154 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:19.017194 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:19.019931 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:19.019932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:19.019933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:19.019941 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:19.021803 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:19.022320 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:19.022321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:19.022322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:19.022385 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:19.022678 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:19.022679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:19.022679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:19.022896 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:19.022942 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:19.023251 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 23:18:19.023343 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:18:19.023659 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:19.010656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:46061 (local address 127.1.4.194:44661)
0924 23:18:19.010812 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:19.010856 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:19.011460 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:19.011857 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:19.011861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:19.012092 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:19.012378 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:19.012385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:19.015427 (+ 3042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:19.015428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:19.016728 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:19.016732 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:19.017344 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:19.017345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:19.017506 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:19.018127 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:19.018137 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:19.019804 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:19.021910 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:19.021918 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:19.021922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:19.022228 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:19.022449 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:19.022450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:19.022450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:19.022598 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:19.022972 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:19.022977 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:19.023105 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 23:18:19.023187 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 23:18:19.023346 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260924 23:18:19.024158 14124 heartbeater.cc:347] Connected to a master server at 127.1.4.254:46061
I20260924 23:18:19.024259 14124 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:19.024430 14124 heartbeater.cc:511] Master 127.1.4.254:46061 requested a full tablet report, sending...
W20260924 23:18:19.024988 14124 heartbeater.cc:644] Failed to heartbeat to 127.1.4.254:46061 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 23:18:19.025053 14124 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:19.025115 14124 heartbeater.cc:511] Master 127.1.4.254:46061 requested a full tablet report, sending...
I20260924 23:18:19.025381 14124 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:19.025458 14124 heartbeater.cc:511] Master 127.1.4.254:46061 requested a full tablet report, sending...
W20260924 23:18:19.025708 14124 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 23:18:19.127003 14127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:19.127354 14127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:19.127475 14127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:19.128997 14127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:19.129051 14127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:19.129091 14127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:18:19.131273 14127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.4.212:34691/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:36019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=39791
--tserver_master_addrs=127.1.4.254:46061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:19.131538 14127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:19.131832 14127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:19.138177 14132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:19.138196 14133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:19.138829 14135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:19.138935 14127 server_base.cc:1034] running on GCE node
I20260924 23:18:19.139242 14127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:19.139537 14127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:19.146909 14127 hybrid_clock.cc:648] HybridClock initialized: now 1790291899146870 us; error 40 us; skew 500 ppm
Sep 24 23:18:19 dist-test-slave-s1hw krb5kdc[12796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291899, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:19.149978 14127 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:18:19.150472 14127 webserver.cc:460] Webserver started at http://127.1.4.195:39791/ using document root <none> and password file <none>
I20260924 23:18:19.150705 14127 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:19.150853 14127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 23:18:19 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291899, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:18:19.175170 14127 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:19.176482 14143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:19.176687 14127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:18:19.176780 14127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "81726cb75cf24e5693d166e92dd308b9"
format_stamp: "Formatted at 2026-09-24 23:18:18 on dist-test-slave-s1hw"
server_key: "f382e8683063cade74a899e08eeb44f5"
server_key_iv: "c9db5970fc05d84fcd02a31a7e8796c7"
server_key_version: "kuduclusterkey@0"
I20260924 23:18:19.177269 14127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:19.194589 14127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:19.196964 14127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:19.197118 14127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:19.197342 14127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:19.197726 14127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:18:19.197785 14127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:19.197814 14127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:18:19.197837 14127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:19.205271 14127 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:36019
I20260924 23:18:19.205755 14127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:18:19.210035 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 14127
Sep 24 23:18:19 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291899, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:19.218653 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:19.206580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:51899 (local address 127.1.4.254:46061)
0924 23:18:19.206842 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:19.206844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:19.206925 (+ 81us) server_negotiation.cc:407] Connection header received
0924 23:18:19.208157 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:19.208159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:19.208172 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:19.208201 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:19.209239 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:19.210019 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:19.212363 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:19.212665 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:19.212708 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:19.215413 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:19.215414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:19.215415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:19.215423 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:19.217099 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:19.217571 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:19.217573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:19.217573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:19.217633 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:19.217896 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:19.217897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:19.217898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:19.218018 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:19.218056 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:19.218475 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 23:18:19.218558 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260924 23:18:19.210491 14257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:36019 every 8 connection(s)
I20260924 23:18:19.219225 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:19.206599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:46061 (local address 127.1.4.195:51899)
0924 23:18:19.206778 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:19.206830 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:19.208025 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:19.208580 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:19.208585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:19.208823 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:19.209113 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:19.209121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:19.210161 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:19.210162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:19.212260 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:19.212263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:19.212887 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:19.212888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:19.213030 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:19.213643 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:19.213653 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:19.215273 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:19.217189 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:19.217196 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:19.217198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:19.217490 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:19.217698 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:19.217698 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:19.217699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:19.217830 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:19.218101 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:19.218108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:19.218301 (+ 193us) client_negotiation.cc:769] Sending connection context
0924 23:18:19.218789 (+ 488us) client_negotiation.cc:240] Negotiation successful
0924 23:18:19.218962 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260924 23:18:19.220217 14258 heartbeater.cc:347] Connected to a master server at 127.1.4.254:46061
I20260924 23:18:19.220337 14258 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:19.220539 14258 heartbeater.cc:511] Master 127.1.4.254:46061 requested a full tablet report, sending...
W20260924 23:18:19.221256 14258 heartbeater.cc:644] Failed to heartbeat to 127.1.4.254:46061 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 23:18:19.221341 14258 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:19.221405 14258 heartbeater.cc:511] Master 127.1.4.254:46061 requested a full tablet report, sending...
I20260924 23:18:19.221763 14258 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:19.221837 14258 heartbeater.cc:511] Master 127.1.4.254:46061 requested a full tablet report, sending...
W20260924 23:18:19.222172 14258 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 23:18:19.225618 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:19.210607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57410 (local address 127.1.4.254:46061)
0924 23:18:19.214661 (+ 4054us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:19.214663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:19.214669 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:18:19.214692 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:19.214693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:19.214706 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:19.214781 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:19.215532 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:19.216552 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:19.217612 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:19.217900 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:19.217952 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:19.221005 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:19.221007 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:19.221007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:19.221014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:19.222939 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:19.223340 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:19.223342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:19.223342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:19.223429 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:19.223633 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:19.223634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:19.223635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:19.223776 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:19.223935 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:19.225474 (+ 1539us) server_negotiation.cc:299] Negotiation successful
0924 23:18:19.225550 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4034,"thread_start_us":47,"threads_started":1}
Sep 24 23:18:19 dist-test-slave-s1hw krb5kdc[12796](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790291899, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:19.862237 13990 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:19.862401 13990 heartbeater.cc:511] Master 127.1.4.254:46061 requested a full tablet report, sending...
I20260924 23:18:19.862782 13792 ts_manager.cc:194] Registered new tserver with Master: 12bfcf8b23734d428c6709a4c4d8599f (127.1.4.193:44999)
I20260924 23:18:19.863536 13792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:52099
I20260924 23:18:20.025883 14124 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:20.026043 14124 heartbeater.cc:511] Master 127.1.4.254:46061 requested a full tablet report, sending...
I20260924 23:18:20.026312 13792 ts_manager.cc:194] Registered new tserver with Master: bf887ee655014b588e56493ea326d169 (127.1.4.194:32947)
I20260924 23:18:20.026850 13792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:44661
Sep 24 23:18:20 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291899, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:18:20.222330 14258 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:20.222478 14258 heartbeater.cc:511] Master 127.1.4.254:46061 requested a full tablet report, sending...
I20260924 23:18:20.222851 13792 ts_manager.cc:194] Registered new tserver with Master: 81726cb75cf24e5693d166e92dd308b9 (127.1.4.195:36019)
I20260924 23:18:20.223490 13792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:51899
I20260924 23:18:20.232570 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:18:20.248014 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:20.233530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57418 (local address 127.1.4.254:46061)
0924 23:18:20.233779 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:20.233781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:20.233928 (+ 147us) server_negotiation.cc:407] Connection header received
0924 23:18:20.233953 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:20.233954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:20.233965 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:20.234020 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:20.238153 (+ 4133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.241451 (+ 3298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.242871 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.243163 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.243218 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:20.244755 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:20.244756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:20.244757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:20.244765 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:20.246541 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:20.246910 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:20.246911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:20.246911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:20.246970 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:20.247131 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:20.247132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:20.247132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:20.247253 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:20.247332 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:20.247837 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 23:18:20.247914 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":61,"threads_started":1}
I20260924 23:18:20.250286 13792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:18:20.252717 13792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:18:20.265012 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:20.256178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:32947 (local address 127.0.0.1:47686)
0924 23:18:20.257041 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:20.257085 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:20.257149 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:20.260116 (+ 2967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:20.260121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:20.260124 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:20.260206 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:20.260208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.261880 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.261881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:20.263216 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:20.263219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.264676 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.264677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:20.264772 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:20.264789 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 23:18:20.264859 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 23:18:20.264920 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":90,"threads_started":1}
I20260924 23:18:20.265483 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:20.256538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.1.4.194:32947)
0924 23:18:20.259513 (+ 2975us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:20.259518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:20.259530 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:18:20.259566 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:20.259568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:20.259584 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:20.259663 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:20.260304 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.261801 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.263315 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.264492 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.264556 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:20.264632 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:20.265345 (+ 713us) server_negotiation.cc:299] Negotiation successful
0924 23:18:20.265406 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2938,"thread_start_us":61,"threads_started":1}
I20260924 23:18:20.266471 14058 tablet_service.cc:1467] Processing CreateTablet for tablet 57bcff7d9d564e788201a2cb192d37c4 (DEFAULT_TABLE table=test-table [id=482aaa6d89c042d29eaad9a64f43ddbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:18:20.266682 14285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:20.256180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:44999 (local address 127.0.0.1:35496)
0924 23:18:20.259789 (+ 3609us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:20.259800 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:20.259883 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:20.260293 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:20.260294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:20.260296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:20.260364 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:20.260366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.261948 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.261948 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:20.265372 (+ 3424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:20.265375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.266461 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.266462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:20.266535 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:20.266539 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:18:20.266577 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 23:18:20.266613 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3592,"mutex_wait_us":2303,"thread_start_us":88,"threads_started":1}
I20260924 23:18:20.266858 14058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bcff7d9d564e788201a2cb192d37c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:20.266929 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:20.256745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35496 (local address 127.1.4.193:44999)
0924 23:18:20.256908 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:20.256912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:20.259855 (+ 2943us) server_negotiation.cc:407] Connection header received
0924 23:18:20.259959 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:20.259963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:20.260031 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:20.260084 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:20.260823 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.261851 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.265473 (+ 3622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.266296 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.266340 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:20.266405 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:20.266818 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 23:18:20.266860 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260924 23:18:20.267534 13905 tablet_service.cc:1467] Processing CreateTablet for tablet 57bcff7d9d564e788201a2cb192d37c4 (DEFAULT_TABLE table=test-table [id=482aaa6d89c042d29eaad9a64f43ddbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:18:20.267823 13905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bcff7d9d564e788201a2cb192d37c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:20.271893 14292 tablet_bootstrap.cc:492] T 57bcff7d9d564e788201a2cb192d37c4 P 12bfcf8b23734d428c6709a4c4d8599f: Bootstrap starting.
I20260924 23:18:20.272575 14292 tablet_bootstrap.cc:654] T 57bcff7d9d564e788201a2cb192d37c4 P 12bfcf8b23734d428c6709a4c4d8599f: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:20.272927 14292 log.cc:824] T 57bcff7d9d564e788201a2cb192d37c4 P 12bfcf8b23734d428c6709a4c4d8599f: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:20.273128 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:20.256413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:36019 (local address 127.0.0.1:43974)
0924 23:18:20.260181 (+ 3768us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:20.260190 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:20.260423 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:20.260708 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:20.260709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:20.260710 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:20.260744 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:20.260746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.263191 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.263192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:20.264273 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:20.264276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.272858 (+ 8582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.272859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:20.272931 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:20.272938 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 23:18:20.272989 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 23:18:20.273039 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3753,"thread_start_us":62,"threads_started":1}
I20260924 23:18:20.273418 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:20.256516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43974 (local address 127.1.4.195:36019)
0924 23:18:20.256618 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:20.256623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:20.260246 (+ 3623us) server_negotiation.cc:407] Connection header received
0924 23:18:20.260518 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:20.260522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:20.260540 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:20.260595 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:20.262096 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.263091 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.271804 (+ 8713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.272676 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.272751 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:20.272814 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:20.273287 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 23:18:20.273335 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":53,"threads_started":1}
I20260924 23:18:20.274047 14292 tablet_bootstrap.cc:492] T 57bcff7d9d564e788201a2cb192d37c4 P 12bfcf8b23734d428c6709a4c4d8599f: No bootstrap required, opened a new log
I20260924 23:18:20.274117 14292 ts_tablet_manager.cc:1397] T 57bcff7d9d564e788201a2cb192d37c4 P 12bfcf8b23734d428c6709a4c4d8599f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:18:20.274230 14192 tablet_service.cc:1467] Processing CreateTablet for tablet 57bcff7d9d564e788201a2cb192d37c4 (DEFAULT_TABLE table=test-table [id=482aaa6d89c042d29eaad9a64f43ddbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:18:20.274549 14192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57bcff7d9d564e788201a2cb192d37c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:20.275508 14291 tablet_bootstrap.cc:492] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169: Bootstrap starting.
I20260924 23:18:20.276086 14291 tablet_bootstrap.cc:654] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:20.276223 14292 raft_consensus.cc:348] T 57bcff7d9d564e788201a2cb192d37c4 P 12bfcf8b23734d428c6709a4c4d8599f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf887ee655014b588e56493ea326d169" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 32947 } } peers { permanent_uuid: "12bfcf8b23734d428c6709a4c4d8599f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44999 } } peers { permanent_uuid: "81726cb75cf24e5693d166e92dd308b9" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36019 } }
I20260924 23:18:20.276391 14292 raft_consensus.cc:374] T 57bcff7d9d564e788201a2cb192d37c4 P 12bfcf8b23734d428c6709a4c4d8599f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:20.276425 14292 raft_consensus.cc:729] T 57bcff7d9d564e788201a2cb192d37c4 P 12bfcf8b23734d428c6709a4c4d8599f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bfcf8b23734d428c6709a4c4d8599f, State: Initialized, Role: FOLLOWER
I20260924 23:18:20.276541 14292 consensus_queue.cc:260] T 57bcff7d9d564e788201a2cb192d37c4 P 12bfcf8b23734d428c6709a4c4d8599f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf887ee655014b588e56493ea326d169" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 32947 } } peers { permanent_uuid: "12bfcf8b23734d428c6709a4c4d8599f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44999 } } peers { permanent_uuid: "81726cb75cf24e5693d166e92dd308b9" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36019 } }
I20260924 23:18:20.276746 14291 log.cc:824] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:20.276796 14292 ts_tablet_manager.cc:1428] T 57bcff7d9d564e788201a2cb192d37c4 P 12bfcf8b23734d428c6709a4c4d8599f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:18:20.276834 13990 heartbeater.cc:503] Master 127.1.4.254:46061 was elected leader, sending a full tablet report...
I20260924 23:18:20.278117 14296 tablet_bootstrap.cc:492] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9: Bootstrap starting.
I20260924 23:18:20.278740 14296 tablet_bootstrap.cc:654] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:20.279079 14296 log.cc:824] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:20.279927 14296 tablet_bootstrap.cc:492] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9: No bootstrap required, opened a new log
I20260924 23:18:20.280043 14296 ts_tablet_manager.cc:1397] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:18:20.282387 14296 raft_consensus.cc:348] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf887ee655014b588e56493ea326d169" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 32947 } } peers { permanent_uuid: "12bfcf8b23734d428c6709a4c4d8599f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44999 } } peers { permanent_uuid: "81726cb75cf24e5693d166e92dd308b9" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36019 } }
I20260924 23:18:20.282637 14296 raft_consensus.cc:374] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:20.282716 14296 raft_consensus.cc:729] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81726cb75cf24e5693d166e92dd308b9, State: Initialized, Role: FOLLOWER
I20260924 23:18:20.282905 14296 consensus_queue.cc:260] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf887ee655014b588e56493ea326d169" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 32947 } } peers { permanent_uuid: "12bfcf8b23734d428c6709a4c4d8599f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44999 } } peers { permanent_uuid: "81726cb75cf24e5693d166e92dd308b9" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36019 } }
I20260924 23:18:20.283245 14296 ts_tablet_manager.cc:1428] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:18:20.283267 14258 heartbeater.cc:503] Master 127.1.4.254:46061 was elected leader, sending a full tablet report...
I20260924 23:18:20.287968 14291 tablet_bootstrap.cc:492] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169: No bootstrap required, opened a new log
I20260924 23:18:20.288050 14291 ts_tablet_manager.cc:1397] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260924 23:18:20.291216 14291 raft_consensus.cc:348] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf887ee655014b588e56493ea326d169" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 32947 } } peers { permanent_uuid: "12bfcf8b23734d428c6709a4c4d8599f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44999 } } peers { permanent_uuid: "81726cb75cf24e5693d166e92dd308b9" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36019 } }
I20260924 23:18:20.291442 14291 raft_consensus.cc:374] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:20.291483 14291 raft_consensus.cc:729] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf887ee655014b588e56493ea326d169, State: Initialized, Role: FOLLOWER
I20260924 23:18:20.291610 14291 consensus_queue.cc:260] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf887ee655014b588e56493ea326d169" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 32947 } } peers { permanent_uuid: "12bfcf8b23734d428c6709a4c4d8599f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44999 } } peers { permanent_uuid: "81726cb75cf24e5693d166e92dd308b9" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36019 } }
I20260924 23:18:20.291901 14291 ts_tablet_manager.cc:1428] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 23:18:20.292043 14124 heartbeater.cc:503] Master 127.1.4.254:46061 was elected leader, sending a full tablet report...
W20260924 23:18:20.333393 13991 tablet.cc:2369] T 57bcff7d9d564e788201a2cb192d37c4 P 12bfcf8b23734d428c6709a4c4d8599f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:18:20.433830 14299 raft_consensus.cc:482] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:18:20.433995 14299 raft_consensus.cc:504] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf887ee655014b588e56493ea326d169" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 32947 } } peers { permanent_uuid: "12bfcf8b23734d428c6709a4c4d8599f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44999 } } peers { permanent_uuid: "81726cb75cf24e5693d166e92dd308b9" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36019 } }
I20260924 23:18:20.434412 14299 leader_election.cc:288] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12bfcf8b23734d428c6709a4c4d8599f (127.1.4.193:44999), 81726cb75cf24e5693d166e92dd308b9 (127.1.4.195:36019)
I20260924 23:18:20.438946 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:20.435083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:59777 (local address 127.1.4.193:44999)
0924 23:18:20.435131 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:20.435133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:20.435393 (+ 260us) server_negotiation.cc:407] Connection header received
0924 23:18:20.435506 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:20.435507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:20.435519 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:20.435545 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:20.435814 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.436518 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.437985 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.438551 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.438598 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:20.438609 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:20.438825 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 23:18:20.438870 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:18:20.439174 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:20.434656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:44999 (local address 127.1.4.194:59777)
0924 23:18:20.435281 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:20.435295 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:20.435426 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:20.435661 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:20.435662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:20.435664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:20.435739 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:20.435741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.436611 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.436611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:20.437878 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:20.437881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.438680 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.438681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:20.438753 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:20.438756 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:18:20.439034 (+ 278us) client_negotiation.cc:240] Negotiation successful
0924 23:18:20.439075 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":125,"threads_started":1}
I20260924 23:18:20.439641 13933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bcff7d9d564e788201a2cb192d37c4" candidate_uuid: "bf887ee655014b588e56493ea326d169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bfcf8b23734d428c6709a4c4d8599f" is_pre_election: true
I20260924 23:18:20.439829 13933 raft_consensus.cc:2454] T 57bcff7d9d564e788201a2cb192d37c4 P 12bfcf8b23734d428c6709a4c4d8599f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf887ee655014b588e56493ea326d169 in term 0.
I20260924 23:18:20.440074 14014 leader_election.cc:302] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bfcf8b23734d428c6709a4c4d8599f, bf887ee655014b588e56493ea326d169; no voters:
I20260924 23:18:20.440207 14299 raft_consensus.cc:2781] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:18:20.440245 14299 raft_consensus.cc:482] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:18:20.440265 14299 raft_consensus.cc:3037] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:20.441458 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:20.435174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:35341 (local address 127.1.4.195:36019)
0924 23:18:20.435210 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:20.435212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:20.435528 (+ 316us) server_negotiation.cc:407] Connection header received
0924 23:18:20.435639 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:20.435641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:20.435654 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:20.435685 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:20.435925 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.436606 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.438247 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.438814 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.438861 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:20.438892 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:20.441329 (+ 2437us) server_negotiation.cc:299] Negotiation successful
0924 23:18:20.441395 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:18:20.441699 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:20.434893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:36019 (local address 127.1.4.194:35341)
0924 23:18:20.435457 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:20.435468 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:20.435563 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:20.435763 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:20.435764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:20.435766 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:20.435839 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:20.435841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.436705 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.436706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:20.438132 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:20.438135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.441165 (+ 3030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.441166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:20.441251 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:20.441255 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:18:20.441586 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 23:18:20.441628 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":57,"threads_started":1}
I20260924 23:18:20.441720 14299 raft_consensus.cc:504] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf887ee655014b588e56493ea326d169" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 32947 } } peers { permanent_uuid: "12bfcf8b23734d428c6709a4c4d8599f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44999 } } peers { permanent_uuid: "81726cb75cf24e5693d166e92dd308b9" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36019 } }
I20260924 23:18:20.441874 14299 leader_election.cc:288] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [CANDIDATE]: Term 1 election: Requested vote from peers 12bfcf8b23734d428c6709a4c4d8599f (127.1.4.193:44999), 81726cb75cf24e5693d166e92dd308b9 (127.1.4.195:36019)
I20260924 23:18:20.442011 13933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bcff7d9d564e788201a2cb192d37c4" candidate_uuid: "bf887ee655014b588e56493ea326d169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bfcf8b23734d428c6709a4c4d8599f"
I20260924 23:18:20.442088 13933 raft_consensus.cc:3037] T 57bcff7d9d564e788201a2cb192d37c4 P 12bfcf8b23734d428c6709a4c4d8599f [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:20.442085 14207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bcff7d9d564e788201a2cb192d37c4" candidate_uuid: "bf887ee655014b588e56493ea326d169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81726cb75cf24e5693d166e92dd308b9" is_pre_election: true
I20260924 23:18:20.442062 14212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57bcff7d9d564e788201a2cb192d37c4" candidate_uuid: "bf887ee655014b588e56493ea326d169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81726cb75cf24e5693d166e92dd308b9"
I20260924 23:18:20.442240 14207 raft_consensus.cc:2454] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf887ee655014b588e56493ea326d169 in term 0.
I20260924 23:18:20.442914 13933 raft_consensus.cc:2454] T 57bcff7d9d564e788201a2cb192d37c4 P 12bfcf8b23734d428c6709a4c4d8599f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf887ee655014b588e56493ea326d169 in term 1.
I20260924 23:18:20.443091 14014 leader_election.cc:302] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bfcf8b23734d428c6709a4c4d8599f, bf887ee655014b588e56493ea326d169; no voters: 81726cb75cf24e5693d166e92dd308b9
I20260924 23:18:20.443215 14299 raft_consensus.cc:2781] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:18:20.443356 14299 raft_consensus.cc:686] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [term 1 LEADER]: Becoming Leader. State: Replica: bf887ee655014b588e56493ea326d169, State: Running, Role: LEADER
I20260924 23:18:20.443630 14299 consensus_queue.cc:237] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf887ee655014b588e56493ea326d169" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 32947 } } peers { permanent_uuid: "12bfcf8b23734d428c6709a4c4d8599f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44999 } } peers { permanent_uuid: "81726cb75cf24e5693d166e92dd308b9" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36019 } }
I20260924 23:18:20.444783 13792 catalog_manager.cc:5495] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf887ee655014b588e56493ea326d169 (127.1.4.194). New cstate: current_term: 1 leader_uuid: "bf887ee655014b588e56493ea326d169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf887ee655014b588e56493ea326d169" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12bfcf8b23734d428c6709a4c4d8599f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81726cb75cf24e5693d166e92dd308b9" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36019 } health_report { overall_health: UNKNOWN } } }
W20260924 23:18:20.462423 14259 tablet.cc:2369] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 23:18:20 dist-test-slave-s1hw krb5kdc[12796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291899, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for HTTP/127.1.4.212@KRBTEST.COM
I20260924 23:18:20.494850 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:20.489467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47702 (local address 127.1.4.194:32947)
0924 23:18:20.489527 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:20.489529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:20.489537 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:18:20.489577 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:20.489578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:20.489603 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:20.489672 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:18:20.490130 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.490883 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.492747 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:20.493058 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:20.493125 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:20.493592 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:18:20.493738 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:18:20.493932 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:20.494669 (+ 737us) server_negotiation.cc:299] Negotiation successful
0924 23:18:20.494698 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 23:18:20.501710 14212 raft_consensus.cc:3037] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:20.501789 13933 raft_consensus.cc:1261] T 57bcff7d9d564e788201a2cb192d37c4 P 12bfcf8b23734d428c6709a4c4d8599f [term 1 FOLLOWER]: Refusing update from remote peer bf887ee655014b588e56493ea326d169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:18:20.502801 14212 raft_consensus.cc:1261] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9 [term 1 FOLLOWER]: Refusing update from remote peer bf887ee655014b588e56493ea326d169: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:18:20.503139 14299 consensus_queue.cc:1035] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bfcf8b23734d428c6709a4c4d8599f" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 44999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:18:20.503533 14299 consensus_queue.cc:1035] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81726cb75cf24e5693d166e92dd308b9" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:18:20.509248 14306 mvcc.cc:205] Tried to move back new op lower bound from 7333035624447098880 to 7333035624218644480. Current Snapshot: MvccSnapshot[applied={T|T < 7333035624447098880 or (T in {7333035624447098880})}]
I20260924 23:18:20.511627 13792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:18:20.511739 13792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:18:20.512794 13792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c09dceb26a9475984319cdeee4d2f93: Sending DeleteTablet for 3 replicas of tablet 57bcff7d9d564e788201a2cb192d37c4
I20260924 23:18:20.513255 13905 tablet_service.cc:1514] Processing DeleteTablet for tablet 57bcff7d9d564e788201a2cb192d37c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:18:20 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:35496
I20260924 23:18:20.513258 14192 tablet_service.cc:1514] Processing DeleteTablet for tablet 57bcff7d9d564e788201a2cb192d37c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:18:20 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:43974
I20260924 23:18:20.513628 14058 tablet_service.cc:1514] Processing DeleteTablet for tablet 57bcff7d9d564e788201a2cb192d37c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:18:20 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:47686
I20260924 23:18:20.514210 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 13830
I20260924 23:18:20.521589 14308 mvcc.cc:205] Tried to move back new op lower bound from 7333035624447098880 to 7333035624218644480. Current Snapshot: MvccSnapshot[applied={T|T < 7333035624447098880}]
I20260924 23:18:20.523468 14316 tablet_replica.cc:330] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9: stopping tablet replica
I20260924 23:18:20.523715 14316 raft_consensus.cc:2229] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:18:20.523857 14316 raft_consensus.cc:2258] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:18:20.526654 14317 tablet_replica.cc:330] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169: stopping tablet replica
I20260924 23:18:20.527469 14317 raft_consensus.cc:2229] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:18:20.531332 14316 ts_tablet_manager.cc:1905] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:18:20.532984 14317 raft_consensus.cc:2258] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:18:20.533413 14316 ts_tablet_manager.cc:1918] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:18:20.533489 14316 log.cc:1196] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/57bcff7d9d564e788201a2cb192d37c4
I20260924 23:18:20.533614 14317 ts_tablet_manager.cc:1905] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:18:20.533740 14316 ts_tablet_manager.cc:1939] T 57bcff7d9d564e788201a2cb192d37c4 P 81726cb75cf24e5693d166e92dd308b9: Deleting consensus metadata
I20260924 23:18:20.534339 13779 catalog_manager.cc:4841] TS 81726cb75cf24e5693d166e92dd308b9 (127.1.4.195:36019): tablet 57bcff7d9d564e788201a2cb192d37c4 (table test-table [id=482aaa6d89c042d29eaad9a64f43ddbf]) successfully deleted
I20260924 23:18:20.539160 14317 ts_tablet_manager.cc:1918] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:18:20.540833 14317 log.cc:1196] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/57bcff7d9d564e788201a2cb192d37c4
I20260924 23:18:20.549111 14317 ts_tablet_manager.cc:1939] T 57bcff7d9d564e788201a2cb192d37c4 P bf887ee655014b588e56493ea326d169: Deleting consensus metadata
I20260924 23:18:20.554759 13777 catalog_manager.cc:4841] TS bf887ee655014b588e56493ea326d169 (127.1.4.194:32947): tablet 57bcff7d9d564e788201a2cb192d37c4 (table test-table [id=482aaa6d89c042d29eaad9a64f43ddbf]) successfully deleted
W20260924 23:18:20.555155 13779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.4.193:44999 (error 108)
I20260924 23:18:20.555620 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 13994
I20260924 23:18:20.567889 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:20.567691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:44999 (local address 127.0.0.1:35498)
0924 23:18:20.567767 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:20.567838 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:44999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 23:18:20.568064 13779 catalog_manager.cc:4584] TS 12bfcf8b23734d428c6709a4c4d8599f (127.1.4.193:44999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57bcff7d9d564e788201a2cb192d37c4: Network error: Client connection negotiation failed: client connection to 127.1.4.193:44999: connect: Connection refused (error 111)
I20260924 23:18:20.575433 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 14127
I20260924 23:18:20.582278 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 13759
I20260924 23:18:20.589768 1043 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 23:18:20.934087 1043 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 23:18:20.934178 1043 mini_ranger.cc:66] Stopping Ranger...
I20260924 23:18:21.268586 1043 mini_ranger.cc:68] Stopped Ranger
2026-09-24 23:18:21.268 UTC [12826] LOG: received smart shutdown request
2026-09-24 23:18:21.272 UTC [12826] LOG: background worker "logical replication launcher" (PID 12835) exited with exit code 1
2026-09-24 23:18:21.272 UTC [12830] LOG: shutting down
2026-09-24 23:18:21.443 UTC [12826] LOG: database system is shut down
2026-09-24T23:18:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46233 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:18:21 dist-test-slave-s1hw krb5kdc[14328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:18:21 dist-test-slave-s1hw krb5kdc[14328](info): set up 2 sockets
Sep 24 23:18:21 dist-test-slave-s1hw krb5kdc[14328](info): commencing operation
krb5kdc: starting...
W20260924 23:18:22.487088 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:18:22 dist-test-slave-s1hw krb5kdc[14328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:18:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:18:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:18:22.602200 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:42701
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:42701,127.1.4.253:44049,127.1.4.252:40681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:22.680850 14341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:22.681053 14341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:22.681075 14341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:22.682144 14341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:18:22.682175 14341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:22.682188 14341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:22.682199 14341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:18:22.682209 14341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:18:22.683673 14341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:42701,127.1.4.253:44049,127.1.4.252:40681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:42701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:22.683869 14341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:22.684038 14341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:22.688859 14346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:22.688915 14341 server_base.cc:1034] running on GCE node
W20260924 23:18:22.688865 14347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:22.688861 14349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:22.689308 14341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:22.689600 14341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:22.690771 14341 hybrid_clock.cc:648] HybridClock initialized: now 1790291902690749 us; error 34 us; skew 500 ppm
Sep 24 23:18:22 dist-test-slave-s1hw krb5kdc[14328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291902, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:22.697320 14341 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:18:22.697772 14341 webserver.cc:460] Webserver started at http://127.1.4.254:42163/ using document root <none> and password file <none>
I20260924 23:18:22.697973 14341 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:22.698011 14341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:22.698104 14341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:18:22.699122 14341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "42a67940032a4f15b864f4cc63bedb3d"
format_stamp: "Formatted at 2026-09-24 23:18:22 on dist-test-slave-s1hw"
server_key: "b040974ef92cadd6419eebb33d101ae6"
server_key_iv: "2a0aef660dc0e4e33df508fcb87d56e1"
server_key_version: "encryptionkey@0"
I20260924 23:18:22.699467 14341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "42a67940032a4f15b864f4cc63bedb3d"
format_stamp: "Formatted at 2026-09-24 23:18:22 on dist-test-slave-s1hw"
server_key: "b040974ef92cadd6419eebb33d101ae6"
server_key_iv: "2a0aef660dc0e4e33df508fcb87d56e1"
server_key_version: "encryptionkey@0"
I20260924 23:18:22.700709 14341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:22.701344 14356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:22.701548 14341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:18:22.701649 14341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "42a67940032a4f15b864f4cc63bedb3d"
format_stamp: "Formatted at 2026-09-24 23:18:22 on dist-test-slave-s1hw"
server_key: "b040974ef92cadd6419eebb33d101ae6"
server_key_iv: "2a0aef660dc0e4e33df508fcb87d56e1"
server_key_version: "encryptionkey@0"
I20260924 23:18:22.701712 14341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:22.722913 14341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:22.725095 14341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:22.725231 14341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:22.729071 14341 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:42701
I20260924 23:18:22.729120 14408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:42701 every 8 connection(s)
I20260924 23:18:22.729454 14341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:18:22.730296 14409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:22.731885 14409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:18:22.736835 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 14341
I20260924 23:18:22.736938 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:18:22.737123 1043 external_mini_cluster.cc:1428] Setting key 9a6abd64d30687fc6bb4c199173a30cc
Sep 24 23:18:22 dist-test-slave-s1hw krb5kdc[14328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291902, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:22.741066 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.732562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42701 (local address 127.0.0.1:56088)
0924 23:18:22.732733 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.732766 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:22.733192 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:22.733445 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:22.733447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:22.733685 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:22.733838 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:22.733841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:22.734834 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:22.734835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:22.735554 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:22.735556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:22.736088 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:22.736089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:22.736138 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:22.736580 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:22.736588 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:22.737984 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:22.739640 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:22.739646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:22.739648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:22.739823 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:22.740086 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:22.740086 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:22.740086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:22.740159 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:22.740589 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:22.740595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:22.740657 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 23:18:22.740759 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 23:18:22.740902 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
I20260924 23:18:22.741092 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.732581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.1.4.254:42701)
0924 23:18:22.732907 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:22.732911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:22.732938 (+ 27us) server_negotiation.cc:407] Connection header received
0924 23:18:22.733329 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:22.733331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:22.733349 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:22.733431 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:22.733936 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:22.734752 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:22.735628 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:22.736013 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:22.736112 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:22.738060 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:22.738067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:22.738068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:22.738074 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:22.739571 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:22.739910 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:22.739914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:22.739914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:22.740020 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:22.740214 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:22.740214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:22.740215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:22.740499 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:22.740586 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:22.740805 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 23:18:22.740992 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":301,"thread_start_us":95,"threads_started":1}
I20260924 23:18:22.741961 14409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:18:22.742290 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.742178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:44049 (local address 127.0.0.1:47664)
0924 23:18:22.742223 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.742267 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:44049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 23:18:22.742420 14360 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.4.253:44049: connect: Connection refused (error 111)
I20260924 23:18:22.742828 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.742730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:44049 (local address 127.0.0.1:47676)
0924 23:18:22.742778 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.742808 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:44049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 23:18:22.742942 14409 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.253:44049: Network error: Client connection negotiation failed: client connection to 127.1.4.253:44049: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.1.4.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.253.keytab.
Entry for principal kudu/127.1.4.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.253.keytab.
I20260924 23:18:22.761587 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.253:44049
--webserver_interface=127.1.4.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.253.keytab
--principal=kudu/127.1.4.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:42701,127.1.4.253:44049,127.1.4.252:40681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:18:22.792176 14409 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } attempt: 1
I20260924 23:18:22.792675 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.792541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:44049 (local address 127.0.0.1:47680)
0924 23:18:22.792595 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.792643 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:44049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 23:18:22.793110 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.793027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:44049 (local address 127.0.0.1:47690)
0924 23:18:22.793068 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.793093 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:44049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 23:18:22.793220 14409 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.253:44049: Network error: Client connection negotiation failed: client connection to 127.1.4.253:44049: connect: Connection refused (error 111)
W20260924 23:18:22.840476 14416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:22.840688 14416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:22.840716 14416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:22.841919 14416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:18:22.841959 14416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:22.841977 14416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:22.841992 14416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:18:22.842010 14416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:18:22.843593 14416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:42701,127.1.4.253:44049,127.1.4.252:40681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.253.keytab
--principal=kudu/127.1.4.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.253:44049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.4.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:22.843843 14416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:22.844043 14416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:18:22.849189 14416 server_base.cc:1034] running on GCE node
W20260924 23:18:22.849159 14424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:22.849174 14421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:22.849154 14422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:22.849758 14416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:22.850101 14416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:22.851257 14416 hybrid_clock.cc:648] HybridClock initialized: now 1790291902851224 us; error 50 us; skew 500 ppm
Sep 24 23:18:22 dist-test-slave-s1hw krb5kdc[14328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291902, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:22.853783 14416 init.cc:377] Logged in from keytab as kudu/127.1.4.253@KRBTEST.COM (short username kudu)
I20260924 23:18:22.854252 14416 webserver.cc:460] Webserver started at http://127.1.4.253:37877/ using document root <none> and password file <none>
I20260924 23:18:22.854462 14416 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:22.854506 14416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:22.854624 14416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:18:22.855748 14416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/data/instance:
uuid: "cece22e1338f426ca648892dd72188ce"
format_stamp: "Formatted at 2026-09-24 23:18:22 on dist-test-slave-s1hw"
server_key: "724cf1a7428a54c9528291b1e7bc1e7d"
server_key_iv: "7e3bec52f6ce638c25d151451aa3f302"
server_key_version: "encryptionkey@0"
I20260924 23:18:22.856082 14416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/wal/instance:
uuid: "cece22e1338f426ca648892dd72188ce"
format_stamp: "Formatted at 2026-09-24 23:18:22 on dist-test-slave-s1hw"
server_key: "724cf1a7428a54c9528291b1e7bc1e7d"
server_key_iv: "7e3bec52f6ce638c25d151451aa3f302"
server_key_version: "encryptionkey@0"
I20260924 23:18:22.857309 14416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:18:22.858037 14431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:22.858213 14416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:22.858294 14416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/wal
uuid: "cece22e1338f426ca648892dd72188ce"
format_stamp: "Formatted at 2026-09-24 23:18:22 on dist-test-slave-s1hw"
server_key: "724cf1a7428a54c9528291b1e7bc1e7d"
server_key_iv: "7e3bec52f6ce638c25d151451aa3f302"
server_key_version: "encryptionkey@0"
I20260924 23:18:22.858347 14416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:22.861435 14409 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } attempt: 2
I20260924 23:18:22.861889 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.861702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:44049 (local address 127.0.0.1:47706)
0924 23:18:22.861812 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.861859 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:44049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 23:18:22.862237 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.862146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:44049 (local address 127.0.0.1:47716)
0924 23:18:22.862188 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.862213 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:44049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 23:18:22.862314 14409 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.253:44049: Network error: Client connection negotiation failed: client connection to 127.1.4.253:44049: connect: Connection refused (error 111)
I20260924 23:18:22.877270 14416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:22.879236 14416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:22.879416 14416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:22.882936 14416 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.253:44049
I20260924 23:18:22.882990 14483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.253:44049 every 8 connection(s)
I20260924 23:18:22.883296 14416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/data/info.pb
I20260924 23:18:22.884132 14484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:22.885661 14484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:18:22.886142 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 14416
I20260924 23:18:22.886231 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-1/wal/instance
I20260924 23:18:22.886438 1043 external_mini_cluster.cc:1428] Setting key 5866db8d68a07ee378a8bb9bcd963457
Sep 24 23:18:22 dist-test-slave-s1hw krb5kdc[14328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291902, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.253@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.4.252@KRBTEST.COM; defaulting to no policy
I20260924 23:18:22.894492 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.886347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.1.4.254:42701)
0924 23:18:22.886399 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:22.886401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:22.886603 (+ 202us) server_negotiation.cc:407] Connection header received
0924 23:18:22.887069 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:22.887070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:22.887086 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:22.887135 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:22.887926 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:22.888475 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:22.889447 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:22.889751 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:22.889838 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:22.892058 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:22.892059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:22.892059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:22.892066 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:22.893362 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:22.893713 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:22.893714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:22.893714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:22.893771 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:22.893988 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:22.893989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:22.893989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:22.894092 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:22.894137 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:22.894362 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 23:18:22.894427 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:18:22.894686 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.886331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42701 (local address 127.0.0.1:56096)
0924 23:18:22.886476 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.886516 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:22.886972 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:22.887203 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:22.887207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:22.887549 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:22.887825 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:22.887831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:22.888571 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:22.888572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:22.889374 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:22.889377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:22.889838 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:22.889839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:22.889933 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:22.890402 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:22.890410 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:22.891970 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:22.893436 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:22.893443 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:22.893445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:22.893654 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:22.893821 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:22.893821 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:22.893822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:22.893938 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:22.894131 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:22.894135 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:22.894262 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 23:18:22.894363 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 23:18:22.894486 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260924 23:18:22.895226 14484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.252@KRBTEST.COM" created.
Sep 24 23:18:22 dist-test-slave-s1hw krb5kdc[14328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291902, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.253@KRBTEST.COM for kudu/127.1.4.253@KRBTEST.COM
I20260924 23:18:22.901996 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.895575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.1.4.253:44049)
0924 23:18:22.895712 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:22.895716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:22.895730 (+ 14us) server_negotiation.cc:407] Connection header received
0924 23:18:22.895744 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:22.895746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:22.895754 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:22.895807 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:22.896065 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:22.896755 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:22.897369 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:22.897655 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:22.897712 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:22.899466 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:22.899473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:22.899474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:22.899479 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:22.900870 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:22.901223 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:22.901225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:22.901226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:22.901271 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:22.901442 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:22.901443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:22.901443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:22.901566 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:22.901623 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:22.901765 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 23:18:22.901823 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260924 23:18:22.902046 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.895512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:44049 (local address 127.0.0.1:47728)
0924 23:18:22.895554 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.895564 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:22.895596 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:22.895820 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:22.895821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:22.895958 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:22.896004 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:22.896006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:22.896825 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:22.896826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:22.897306 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:22.897308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:22.897715 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:22.897716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:22.897774 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:22.898099 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:22.898105 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:22.899338 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:22.900948 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:22.900949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:22.900949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:22.901152 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:22.901323 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:22.901324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:22.901324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:22.901389 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:22.901621 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:22.901622 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:22.901667 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 23:18:22.901745 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 23:18:22.901876 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":12}
I20260924 23:18:22.902644 14484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:18:22.902990 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.902868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:40681 (local address 127.0.0.1:57632)
0924 23:18:22.902915 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.902967 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 23:18:22.903160 14432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.4.252:40681: connect: Connection refused (error 111)
I20260924 23:18:22.903586 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.903511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:40681 (local address 127.0.0.1:57644)
0924 23:18:22.903548 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.903571 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 23:18:22.903671 14484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.252:40681: Network error: Client connection negotiation failed: client connection to 127.1.4.252:40681: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.252.keytab.
Entry for principal kudu/127.1.4.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.252.keytab.
I20260924 23:18:22.910708 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.252:40681
--webserver_interface=127.1.4.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.252.keytab
--principal=kudu/127.1.4.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:42701,127.1.4.253:44049,127.1.4.252:40681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:18:22.952894 14484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } attempt: 1
I20260924 23:18:22.953356 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.953241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:40681 (local address 127.0.0.1:57654)
0924 23:18:22.953293 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.953325 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 23:18:22.953446 14409 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } attempt: 3
I20260924 23:18:22.953716 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.953584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:40681 (local address 127.0.0.1:57666)
0924 23:18:22.953633 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.953675 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 23:18:22.953976 14484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.252:40681: Network error: Client connection negotiation failed: client connection to 127.1.4.252:40681: connect: Connection refused (error 111)
Sep 24 23:18:22 dist-test-slave-s1hw krb5kdc[14328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291902, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for kudu/127.1.4.253@KRBTEST.COM
I20260924 23:18:22.959838 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.953726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47740 (local address 127.1.4.253:44049)
0924 23:18:22.953768 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:22.953770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:22.953779 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:18:22.953863 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:22.953864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:22.953876 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:22.954247 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:22.954277 (+ 30us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:22.954762 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:22.955452 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:22.955659 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:22.955713 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:22.957634 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:22.957636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:22.957636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:22.957642 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:22.958850 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:22.959206 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:22.959206 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:22.959207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:22.959252 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:22.959431 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:22.959432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:22.959432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:22.959532 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:22.959580 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:22.959720 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 23:18:22.959779 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:18:22.959895 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.953691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:44049 (local address 127.0.0.1:47740)
0924 23:18:22.953737 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.953747 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:22.953790 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:22.953955 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:22.953956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:22.954126 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:22.954191 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:22.954194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:22.954839 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:22.954841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:22.955359 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:22.955360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:22.955719 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:22.955720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:22.955794 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:22.956189 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:22.956195 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:22.957556 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:22.958930 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:22.958931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:22.958932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:22.959135 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:22.959301 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:22.959302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:22.959302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:22.959382 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:22.959583 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:22.959584 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:22.959637 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 23:18:22.959701 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 23:18:22.959803 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 23:18:22.960278 14409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:18:22.960593 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.960495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:40681 (local address 127.0.0.1:57680)
0924 23:18:22.960539 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.960569 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 23:18:22.960888 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.960806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:40681 (local address 127.0.0.1:57694)
0924 23:18:22.960846 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.960871 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 23:18:22.960968 14409 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.252:40681: Network error: Client connection negotiation failed: client connection to 127.1.4.252:40681: connect: Connection refused (error 111)
W20260924 23:18:22.990459 14491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:22.990651 14491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:22.990670 14491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:22.991755 14491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:18:22.991786 14491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:22.991799 14491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:22.991811 14491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:18:22.991827 14491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:18:22.992115 14409 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } attempt: 1
I20260924 23:18:22.992543 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.992407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:40681 (local address 127.0.0.1:57704)
0924 23:18:22.992469 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.992510 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 23:18:22.992858 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:22.992764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:40681 (local address 127.0.0.1:57716)
0924 23:18:22.992805 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:22.992835 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 23:18:22.992966 14409 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.252:40681: Network error: Client connection negotiation failed: client connection to 127.1.4.252:40681: connect: Connection refused (error 111)
I20260924 23:18:22.993496 14491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:42701,127.1.4.253:44049,127.1.4.252:40681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.252.keytab
--principal=kudu/127.1.4.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.252:40681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.4.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:22.993714 14491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:22.993912 14491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:22.998535 14497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:22.998551 14491 server_base.cc:1034] running on GCE node
W20260924 23:18:22.998536 14499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:22.998551 14496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:22.999042 14491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:22.999330 14491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:23.000499 14491 hybrid_clock.cc:648] HybridClock initialized: now 1790291903000483 us; error 28 us; skew 500 ppm
Sep 24 23:18:22 dist-test-slave-s1hw krb5kdc[14328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291902, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:23.002777 14491 init.cc:377] Logged in from keytab as kudu/127.1.4.252@KRBTEST.COM (short username kudu)
I20260924 23:18:23.003185 14491 webserver.cc:460] Webserver started at http://127.1.4.252:40085/ using document root <none> and password file <none>
I20260924 23:18:23.003392 14491 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:23.003441 14491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:23.003535 14491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:18:23.004535 14491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/data/instance:
uuid: "a481bd2063da4ec1978cd6ce1522279c"
format_stamp: "Formatted at 2026-09-24 23:18:23 on dist-test-slave-s1hw"
server_key: "fb381229a012bc8b625487d67baf540e"
server_key_iv: "60fb565c20de71085628306948f1e9ee"
server_key_version: "encryptionkey@0"
I20260924 23:18:23.004824 14491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/wal/instance:
uuid: "a481bd2063da4ec1978cd6ce1522279c"
format_stamp: "Formatted at 2026-09-24 23:18:23 on dist-test-slave-s1hw"
server_key: "fb381229a012bc8b625487d67baf540e"
server_key_iv: "60fb565c20de71085628306948f1e9ee"
server_key_version: "encryptionkey@0"
I20260924 23:18:23.005995 14491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 23:18:23.006637 14506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:23.006808 14491 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:18:23.006894 14491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/wal
uuid: "a481bd2063da4ec1978cd6ce1522279c"
format_stamp: "Formatted at 2026-09-24 23:18:23 on dist-test-slave-s1hw"
server_key: "fb381229a012bc8b625487d67baf540e"
server_key_iv: "60fb565c20de71085628306948f1e9ee"
server_key_version: "encryptionkey@0"
I20260924 23:18:23.006951 14491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:23.022166 14491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:23.022125 14484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } attempt: 2
I20260924 23:18:23.022563 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.022433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:40681 (local address 127.0.0.1:57726)
0924 23:18:23.022486 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:23.022526 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 23:18:23.022893 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.022792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:40681 (local address 127.0.0.1:57734)
0924 23:18:23.022840 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:23.022872 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 23:18:23.022969 14484 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.252:40681: Network error: Client connection negotiation failed: client connection to 127.1.4.252:40681: connect: Connection refused (error 111)
I20260924 23:18:23.024430 14491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:23.024565 14491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:23.028369 14491 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.252:40681
I20260924 23:18:23.028435 14558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.252:40681 every 8 connection(s)
I20260924 23:18:23.028895 14491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/data/info.pb
I20260924 23:18:23.029515 14559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:23.031036 14559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:18:23.034968 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 14491
I20260924 23:18:23.035065 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291755455739-1043-0/minicluster-data/master-2/wal/instance
I20260924 23:18:23.035223 1043 external_mini_cluster.cc:1428] Setting key d11238038a3896a1487eadfc51857e24
Sep 24 23:18:23 dist-test-slave-s1hw krb5kdc[14328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291902, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.252@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:23.039460 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.031741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56098 (local address 127.1.4.254:42701)
0924 23:18:23.031806 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:23.031808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:23.031976 (+ 168us) server_negotiation.cc:407] Connection header received
0924 23:18:23.032408 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:23.032408 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:23.032417 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:23.032462 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:23.033016 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.033474 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.034377 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.034552 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.034595 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:23.036845 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:23.036846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:23.036847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:23.036854 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:23.038143 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.038484 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.038485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.038485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.038544 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.038786 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.038787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.038787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.038887 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:23.038932 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:23.039249 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 23:18:23.039327 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 23:18:23.039539 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.031701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:42701 (local address 127.0.0.1:56098)
0924 23:18:23.031858 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:23.031902 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:23.032330 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:23.032521 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:23.032524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:23.032761 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:23.032955 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:23.032960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.033537 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.033538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:23.034304 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:23.034306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.034605 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.034605 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:23.034698 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:23.035183 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:23.035191 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:23.036763 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:23.038216 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:23.038221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:23.038223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:23.038425 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:23.038597 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:23.038598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:23.038598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:23.038722 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:23.038962 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:23.038969 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:23.039129 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 23:18:23.039212 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 23:18:23.039323 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260924 23:18:23.040031 14559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:18:23 dist-test-slave-s1hw krb5kdc[14328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.252@KRBTEST.COM
I20260924 23:18:23.043280 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.035586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.1.4.252:40681)
0924 23:18:23.035724 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:23.035727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:23.035792 (+ 65us) server_negotiation.cc:407] Connection header received
0924 23:18:23.035896 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:23.035900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:23.035921 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:23.036008 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:23.036521 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.037392 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.038156 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.038516 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.038579 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:23.040938 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:23.040945 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:23.040945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:23.040951 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:23.042250 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.042566 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.042568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.042569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.042619 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.042797 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.042798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.042798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.042949 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:23.043060 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:23.043155 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 23:18:23.043209 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
Sep 24 23:18:23 dist-test-slave-s1hw krb5kdc[14328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291902, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.252@KRBTEST.COM for kudu/127.1.4.253@KRBTEST.COM
I20260924 23:18:23.046285 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.040668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:44049 (local address 127.0.0.1:47750)
0924 23:18:23.040709 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:23.040718 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:23.040747 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:23.040903 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:23.040904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:23.041020 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:23.041059 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:23.041060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.041624 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.041625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:23.042170 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:23.042172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.042486 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.042487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:23.042549 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:23.042865 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:23.042869 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:23.044028 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:23.045450 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:23.045451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:23.045451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:23.045596 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:23.045755 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:23.045755 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:23.045756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:23.045806 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:23.046010 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:23.046010 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:23.046049 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 23:18:23.046113 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 23:18:23.046194 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 23:18:23.046290 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.040752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47750 (local address 127.1.4.253:44049)
0924 23:18:23.040808 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:23.040810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:23.040817 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:18:23.040839 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:23.040840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:23.040854 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:23.040895 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:23.041121 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.041564 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.042238 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.042434 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.042485 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:23.044098 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:23.044099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:23.044100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:23.044106 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:23.045384 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.045654 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.045655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.045655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.045705 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.045858 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.045859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.045859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.045960 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:23.046011 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:23.046141 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 23:18:23.046206 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 23:18:23.046746 14559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:18:23 dist-test-slave-s1hw krb5kdc[14328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
Sep 24 23:18:23 dist-test-slave-s1hw krb5kdc[14328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291902, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.252@KRBTEST.COM for kudu/127.1.4.252@KRBTEST.COM
I20260924 23:18:23.050810 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.044055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56110 (local address 127.1.4.254:42701)
0924 23:18:23.044087 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:23.044089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:23.044136 (+ 47us) server_negotiation.cc:407] Connection header received
0924 23:18:23.044201 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:23.044202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:23.044210 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:23.044252 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:23.044645 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.045078 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.045683 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.045878 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.045929 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:23.048160 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:23.048161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:23.048161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:23.048168 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:23.049488 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.049764 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.049765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.049765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.049812 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.049972 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.049973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.049973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.050067 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:23.050607 (+ 540us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:23.050639 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 23:18:23.050711 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 23:18:23.052927 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.046975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:40681 (local address 127.0.0.1:57746)
0924 23:18:23.047020 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:23.047030 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:23.047067 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:23.047170 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:23.047171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:23.047295 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:23.047328 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:23.047329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.048108 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.048109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:23.048715 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:23.048717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.049193 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.049194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:23.049252 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:23.049554 (+ 302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:23.049559 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:23.050621 (+ 1062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:23.052091 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:23.052092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:23.052092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:23.052231 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:23.052379 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:23.052380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:23.052380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:23.052428 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:23.052615 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:23.052616 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:23.052649 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 23:18:23.052708 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 23:18:23.052794 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 23:18:23.052971 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.047024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57746 (local address 127.1.4.252:40681)
0924 23:18:23.047057 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:23.047058 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:23.047066 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:18:23.047110 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:23.047111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:23.047120 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:23.047165 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:23.047437 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.048017 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.048789 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.049115 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.049183 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:23.050691 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:23.050693 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:23.050693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:23.050699 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:23.052024 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.052284 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.052285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.052286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.052329 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.052475 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.052476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.052476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.052568 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:23.052616 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:23.052743 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 23:18:23.052840 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":17}
I20260924 23:18:23.054771 14559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c: Bootstrap starting.
Sep 24 23:18:23 dist-test-slave-s1hw krb5kdc[14328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.253@KRBTEST.COM
I20260924 23:18:23.055796 14559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:23.056133 14559 log.cc:824] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:23.056923 14559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c: No bootstrap required, opened a new log
I20260924 23:18:23.058646 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.051434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.1.4.253:44049)
0924 23:18:23.051478 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:23.051480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:23.051489 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:18:23.051697 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:23.051698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:23.051708 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:23.051778 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:23.052205 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.052646 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.053570 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.053834 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.053915 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:23.055880 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:23.055881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:23.055881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:23.055888 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:23.057364 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.057689 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.057690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.057690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.057733 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.057900 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.057900 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.057901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.057985 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:23.058493 (+ 508us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:23.058526 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 23:18:23.058584 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:18:23.058653 14559 raft_consensus.cc:348] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } }
I20260924 23:18:23.058805 14559 raft_consensus.cc:374] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:23.058835 14559 raft_consensus.cc:729] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a481bd2063da4ec1978cd6ce1522279c, State: Initialized, Role: FOLLOWER
I20260924 23:18:23.058970 14559 consensus_queue.cc:260] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } }
I20260924 23:18:23.059211 14565 sys_catalog.cc:455] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } } }
I20260924 23:18:23.059264 14559 sys_catalog.cc:564] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:18:23.059311 14565 sys_catalog.cc:458] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:18:23.059962 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 23:18:23.061053 14576 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:18:23.061162 14576 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:18:23.061769 14491 master_runner.cc:186] Error getting master registration for 127.1.4.254:42701: OK, instance_id { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" instance_seqno: 1790291902727964 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 23:18:23.064070 14491 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 23:18:23.068214 14409 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } attempt: 2
I20260924 23:18:23.070241 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.061557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.1.4.252:40681)
0924 23:18:23.061596 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:23.061598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:23.061859 (+ 261us) server_negotiation.cc:407] Connection header received
0924 23:18:23.061940 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:23.061941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:23.061949 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:23.061976 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:23.062555 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.063244 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.064794 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.065074 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.065141 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:23.066635 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:23.066637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:23.066637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:23.066643 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:23.068127 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.069262 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.069264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.069264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.069319 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.069495 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.069496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.069496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.069647 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:23.069742 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:23.070084 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 23:18:23.070156 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:18:23.070338 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.061607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.1.4.253:44049)
0924 23:18:23.061645 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:23.061647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:23.063577 (+ 1930us) server_negotiation.cc:407] Connection header received
0924 23:18:23.063662 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:23.063663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:23.063672 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:23.063697 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:23.064380 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.065072 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.066040 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.066318 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.066356 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:23.067763 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:23.067764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:23.067764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:23.067770 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:23.069171 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.069539 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.069540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.069540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.069592 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.069776 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.069777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.069777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.069902 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:23.069979 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:23.070172 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 23:18:23.070245 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:18:23.072042 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.061246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.1.4.254:42701)
0924 23:18:23.061286 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:23.061288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:23.061587 (+ 299us) server_negotiation.cc:407] Connection header received
0924 23:18:23.061687 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:23.061687 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:23.061698 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:23.061734 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:23.062474 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.063219 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.064817 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.065095 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.065129 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:23.067033 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:23.067035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:23.067035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:23.067042 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:23.069038 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.071344 (+ 2306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.071346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.071346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.071421 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.071575 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.071576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.071576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.071682 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:23.071765 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:23.071889 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 23:18:23.071960 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 23:18:23 dist-test-slave-s1hw krb5kdc[14328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291902, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for kudu/127.1.4.252@KRBTEST.COM
I20260924 23:18:23.076025 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.068862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.1.4.252:40681)
0924 23:18:23.069221 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:23.069223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:23.069229 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:18:23.069247 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:23.069247 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:23.069256 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:23.069579 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:23.069616 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.070780 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.071714 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.072022 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.072082 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:23.073768 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:23.073769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:23.073769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:23.073775 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:23.075026 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.075385 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.075386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.075386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.075443 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.075624 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.075625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.075625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.075715 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:23.075759 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:23.075893 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 23:18:23.075945 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":88,"threads_started":1}
I20260924 23:18:23.076053 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.068465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:40681 (local address 127.0.0.1:57772)
0924 23:18:23.068720 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:23.068731 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:23.068768 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:23.069346 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:23.069347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:23.069480 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:23.069520 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:23.069522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.070866 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.070867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:23.071624 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:23.071627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.072098 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.072099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:23.072160 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:23.072483 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:23.072487 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:23.073680 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:23.075095 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:23.075096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:23.075097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:23.075293 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:23.075503 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:23.075504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:23.075504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:23.075560 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:23.075763 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:23.075764 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:23.075804 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 23:18:23.075876 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 23:18:23.075960 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
I20260924 23:18:23.077361 14409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d: Bootstrap starting.
I20260924 23:18:23.077980 14409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:23.078279 14409 log.cc:824] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:23.078953 14409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d: No bootstrap required, opened a new log
I20260924 23:18:23.080370 14409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } }
I20260924 23:18:23.080474 14409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:23.080495 14409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a67940032a4f15b864f4cc63bedb3d, State: Initialized, Role: FOLLOWER
I20260924 23:18:23.080600 14409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } }
I20260924 23:18:23.080817 14587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } } }
I20260924 23:18:23.080878 14409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:18:23.080904 14587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d [sys.catalog]: This master's current role is: FOLLOWER
W20260924 23:18:23.082391 14598 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:18:23.082442 14598 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:18:23.083212 14341 master_runner.cc:186] Error getting master registration for 127.1.4.253:44049: OK, instance_id { permanent_uuid: "cece22e1338f426ca648892dd72188ce" instance_seqno: 1790291902881955 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 23:18:23.085841 14341 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 23:18:23.114120 14484 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } attempt: 3
Sep 24 23:18:23 dist-test-slave-s1hw krb5kdc[14328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291902, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.253@KRBTEST.COM for kudu/127.1.4.252@KRBTEST.COM
I20260924 23:18:23.122282 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.114455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:40681 (local address 127.0.0.1:57786)
0924 23:18:23.114533 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:23.114546 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:23.114602 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:23.114761 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:23.114761 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:23.114962 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:23.115013 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:23.115015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.115939 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.115940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:23.116765 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:23.116767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.117241 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.117242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:23.117302 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:23.117791 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:23.117796 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:23.119409 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:23.121224 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:23.121225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:23.121226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:23.121423 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:23.121644 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:23.121644 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:23.121644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:23.121704 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:23.121969 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:23.121970 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:23.122017 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 23:18:23.122082 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 23:18:23.122179 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 23:18:23.122309 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:23.114540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.1.4.252:40681)
0924 23:18:23.114583 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:23.114585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:23.114595 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:18:23.114676 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:23.114677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:23.114687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:23.114746 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:23.115088 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.115863 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.116863 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:23.117168 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:23.117238 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:23.119498 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:23.119499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:23.119499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:23.119507 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:23.121133 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.121497 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.121499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.121499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.121573 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:23.121763 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:23.121764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:23.121765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:23.121900 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:23.121975 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:23.122119 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 23:18:23.122217 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:18:23.124290 14484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce: Bootstrap starting.
I20260924 23:18:23.125016 14484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:23.125295 14484 log.cc:824] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:23.126185 14484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce: No bootstrap required, opened a new log
I20260924 23:18:23.128281 14484 raft_consensus.cc:348] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } }
I20260924 23:18:23.128453 14484 raft_consensus.cc:374] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:23.128526 14484 raft_consensus.cc:729] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cece22e1338f426ca648892dd72188ce, State: Initialized, Role: FOLLOWER
I20260924 23:18:23.128684 14484 consensus_queue.cc:260] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } }
I20260924 23:18:23.129071 14484 sys_catalog.cc:564] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:18:23.129048 14601 sys_catalog.cc:455] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } } }
I20260924 23:18:23.129259 14601 sys_catalog.cc:458] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [sys.catalog]: This master's current role is: FOLLOWER
W20260924 23:18:23.130007 14437 tablet.cc:2369] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:18:23.130941 14612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:18:23.131067 14612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:18:23.133052 14416 master_runner.cc:263] No leader master found from master cece22e1338f426ca648892dd72188ce
I20260924 23:18:23.133097 14416 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 23:18:23.225975 14362 tablet.cc:2369] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:18:23.275235 14512 tablet.cc:2369] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:18:23.463316 14601 raft_consensus.cc:482] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:18:23.463462 14601 raft_consensus.cc:504] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } }
I20260924 23:18:23.463685 14601 leader_election.cc:288] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42a67940032a4f15b864f4cc63bedb3d (127.1.4.254:42701), a481bd2063da4ec1978cd6ce1522279c (127.1.4.252:40681)
I20260924 23:18:23.463989 14533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cece22e1338f426ca648892dd72188ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a481bd2063da4ec1978cd6ce1522279c" is_pre_election: true
I20260924 23:18:23.464003 14383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cece22e1338f426ca648892dd72188ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a67940032a4f15b864f4cc63bedb3d" is_pre_election: true
I20260924 23:18:23.464159 14383 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cece22e1338f426ca648892dd72188ce in term 0.
I20260924 23:18:23.464161 14533 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cece22e1338f426ca648892dd72188ce in term 0.
I20260924 23:18:23.464358 14435 leader_election.cc:302] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42a67940032a4f15b864f4cc63bedb3d, cece22e1338f426ca648892dd72188ce; no voters:
I20260924 23:18:23.464516 14601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:18:23.464586 14601 raft_consensus.cc:482] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:18:23.464610 14601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:23.465359 14601 raft_consensus.cc:504] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } }
I20260924 23:18:23.465473 14601 leader_election.cc:288] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [CANDIDATE]: Term 1 election: Requested vote from peers 42a67940032a4f15b864f4cc63bedb3d (127.1.4.254:42701), a481bd2063da4ec1978cd6ce1522279c (127.1.4.252:40681)
I20260924 23:18:23.465621 14533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cece22e1338f426ca648892dd72188ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a481bd2063da4ec1978cd6ce1522279c"
I20260924 23:18:23.465631 14383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cece22e1338f426ca648892dd72188ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a67940032a4f15b864f4cc63bedb3d"
I20260924 23:18:23.465706 14533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:23.465708 14383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:23.466384 14383 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cece22e1338f426ca648892dd72188ce in term 1.
I20260924 23:18:23.466387 14533 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cece22e1338f426ca648892dd72188ce in term 1.
I20260924 23:18:23.466562 14432 leader_election.cc:302] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a481bd2063da4ec1978cd6ce1522279c, cece22e1338f426ca648892dd72188ce; no voters:
I20260924 23:18:23.466658 14601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:18:23.466771 14601 raft_consensus.cc:686] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [term 1 LEADER]: Becoming Leader. State: Replica: cece22e1338f426ca648892dd72188ce, State: Running, Role: LEADER
I20260924 23:18:23.466848 14601 consensus_queue.cc:237] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } }
I20260924 23:18:23.467206 14614 sys_catalog.cc:455] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader cece22e1338f426ca648892dd72188ce. Latest consensus state: current_term: 1 leader_uuid: "cece22e1338f426ca648892dd72188ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } } }
I20260924 23:18:23.467281 14614 sys_catalog.cc:458] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [sys.catalog]: This master's current role is: LEADER
I20260924 23:18:23.467504 14616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:18:23.467669 14616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:18:23.468915 14533 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c [term 1 FOLLOWER]: Refusing update from remote peer cece22e1338f426ca648892dd72188ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:18:23.468991 14383 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d [term 1 FOLLOWER]: Refusing update from remote peer cece22e1338f426ca648892dd72188ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:18:23.469266 14601 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:18:23.469383 14614 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:18:23.470414 14587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cece22e1338f426ca648892dd72188ce. Latest consensus state: current_term: 1 leader_uuid: "cece22e1338f426ca648892dd72188ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } } }
I20260924 23:18:23.470512 14587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:18:23.470469 14565 sys_catalog.cc:455] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cece22e1338f426ca648892dd72188ce. Latest consensus state: current_term: 1 leader_uuid: "cece22e1338f426ca648892dd72188ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } } }
I20260924 23:18:23.470547 14565 sys_catalog.cc:458] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:18:23.471208 14617 mvcc.cc:205] Tried to move back new op lower bound from 7333035636606750720 to 7333035636600770560. Current Snapshot: MvccSnapshot[applied={T|T < 7333035636606750720}]
I20260924 23:18:23.471294 14614 sys_catalog.cc:455] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cece22e1338f426ca648892dd72188ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } } }
I20260924 23:18:23.471406 14614 sys_catalog.cc:458] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [sys.catalog]: This master's current role is: LEADER
I20260924 23:18:23.471449 14614 sys_catalog.cc:455] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cece22e1338f426ca648892dd72188ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } } }
I20260924 23:18:23.471484 14614 sys_catalog.cc:458] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce [sys.catalog]: This master's current role is: LEADER
I20260924 23:18:23.471503 14587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cece22e1338f426ca648892dd72188ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } } }
I20260924 23:18:23.471565 14587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:18:23.471638 14619 mvcc.cc:205] Tried to move back new op lower bound from 7333035636606750720 to 7333035636600770560. Current Snapshot: MvccSnapshot[applied={T|T < 7333035636606750720}]
I20260924 23:18:23.471618 14565 sys_catalog.cc:455] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cece22e1338f426ca648892dd72188ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42a67940032a4f15b864f4cc63bedb3d" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 42701 } } peers { permanent_uuid: "cece22e1338f426ca648892dd72188ce" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 44049 } } peers { permanent_uuid: "a481bd2063da4ec1978cd6ce1522279c" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 40681 } } }
I20260924 23:18:23.471688 14565 sys_catalog.cc:458] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:18:23.471720 14616 catalog_manager.cc:1347] Generated new cluster ID: ade8267c4dce4c9f890fd19bad32322b
I20260924 23:18:23.471763 14616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:18:23.480412 14616 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:18:23.480939 14616 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:18:23.486876 14616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cece22e1338f426ca648892dd72188ce: Generated new TSK 0
I20260924 23:18:23.487152 14616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:18:24.061664 14576 catalog_manager.cc:1259] Loaded cluster ID: ade8267c4dce4c9f890fd19bad32322b
I20260924 23:18:24.061751 14576 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c: loading cluster ID for follower catalog manager: success
I20260924 23:18:24.062506 14576 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c: acquiring CA information for follower catalog manager: success
I20260924 23:18:24.062741 14576 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a481bd2063da4ec1978cd6ce1522279c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 23:18:24.082885 14598 catalog_manager.cc:1259] Loaded cluster ID: ade8267c4dce4c9f890fd19bad32322b
I20260924 23:18:24.082960 14598 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d: loading cluster ID for follower catalog manager: success
I20260924 23:18:24.083755 14598 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d: acquiring CA information for follower catalog manager: success
I20260924 23:18:24.084025 14598 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 42a67940032a4f15b864f4cc63bedb3d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 23:18:24.553254 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 14341
I20260924 23:18:24.557775 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 14416
I20260924 23:18:24.561831 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 14491
2026-09-24T23:18:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3030 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:18:24 dist-test-slave-s1hw krb5kdc[14632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:18:24 dist-test-slave-s1hw krb5kdc[14632](info): set up 2 sockets
Sep 24 23:18:24 dist-test-slave-s1hw krb5kdc[14632](info): commencing operation
krb5kdc: starting...
W20260924 23:18:25.507128 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:18:25 dist-test-slave-s1hw krb5kdc[14632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:18:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:18:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:18:25.618299 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:34415
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:34415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:25.699868 14645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:25.700071 14645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:25.700093 14645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:25.701223 14645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:18:25.701265 14645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:25.701278 14645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:25.701289 14645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:18:25.701299 14645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:18:25.702764 14645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:34415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:34415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:25.702986 14645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:25.703195 14645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:25.707676 14650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:25.707757 14653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:25.707676 14651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:25.707988 14645 server_base.cc:1034] running on GCE node
I20260924 23:18:25.708213 14645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:25.708496 14645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:25.709628 14645 hybrid_clock.cc:648] HybridClock initialized: now 1790291905709614 us; error 25 us; skew 500 ppm
Sep 24 23:18:25 dist-test-slave-s1hw krb5kdc[14632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291905, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:25.711982 14645 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:18:25.712416 14645 webserver.cc:460] Webserver started at http://127.1.4.254:38175/ using document root <none> and password file <none>
I20260924 23:18:25.712604 14645 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:25.712649 14645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:25.712770 14645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:18:25.713773 14645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "548b172f29eb4937b86835dbd7ebbc9c"
format_stamp: "Formatted at 2026-09-24 23:18:25 on dist-test-slave-s1hw"
server_key: "9b15eec63858c3dc826d0564b82aad7c"
server_key_iv: "fb2012549766988b9d9dbc2e5d8c0817"
server_key_version: "encryptionkey@0"
I20260924 23:18:25.714095 14645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "548b172f29eb4937b86835dbd7ebbc9c"
format_stamp: "Formatted at 2026-09-24 23:18:25 on dist-test-slave-s1hw"
server_key: "9b15eec63858c3dc826d0564b82aad7c"
server_key_iv: "fb2012549766988b9d9dbc2e5d8c0817"
server_key_version: "encryptionkey@0"
I20260924 23:18:25.715271 14645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:18:25.715978 14660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:25.716125 14645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:18:25.716205 14645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "548b172f29eb4937b86835dbd7ebbc9c"
format_stamp: "Formatted at 2026-09-24 23:18:25 on dist-test-slave-s1hw"
server_key: "9b15eec63858c3dc826d0564b82aad7c"
server_key_iv: "fb2012549766988b9d9dbc2e5d8c0817"
server_key_version: "encryptionkey@0"
I20260924 23:18:25.716264 14645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:25.740139 14645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:25.742244 14645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:25.742378 14645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:25.746094 14645 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:34415
I20260924 23:18:25.746146 14712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:34415 every 8 connection(s)
I20260924 23:18:25.746457 14645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:18:25.747252 14713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:25.749511 14713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c: Bootstrap starting.
I20260924 23:18:25.750051 14713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:25.750296 14713 log.cc:824] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:25.750931 14713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c: No bootstrap required, opened a new log
I20260924 23:18:25.752513 14713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548b172f29eb4937b86835dbd7ebbc9c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 34415 } }
I20260924 23:18:25.752646 14713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:25.752667 14713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548b172f29eb4937b86835dbd7ebbc9c, State: Initialized, Role: FOLLOWER
I20260924 23:18:25.752802 14713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548b172f29eb4937b86835dbd7ebbc9c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 34415 } }
I20260924 23:18:25.752895 14713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:18:25.752934 14713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:18:25.752982 14713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:25.753000 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 14645
I20260924 23:18:25.753086 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:18:25.753263 1043 external_mini_cluster.cc:1428] Setting key b13fc4ec1272e9f6a8472f4e92008756
I20260924 23:18:25.753582 14713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548b172f29eb4937b86835dbd7ebbc9c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 34415 } }
I20260924 23:18:25.753680 14713 leader_election.cc:302] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 548b172f29eb4937b86835dbd7ebbc9c; no voters:
I20260924 23:18:25.753824 14713 leader_election.cc:288] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:18:25.753958 14718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:18:25.754076 14713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:18:25.754114 14718 raft_consensus.cc:686] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [term 1 LEADER]: Becoming Leader. State: Replica: 548b172f29eb4937b86835dbd7ebbc9c, State: Running, Role: LEADER
I20260924 23:18:25.754215 14718 consensus_queue.cc:237] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548b172f29eb4937b86835dbd7ebbc9c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 34415 } }
I20260924 23:18:25.754720 14720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 548b172f29eb4937b86835dbd7ebbc9c. Latest consensus state: current_term: 1 leader_uuid: "548b172f29eb4937b86835dbd7ebbc9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548b172f29eb4937b86835dbd7ebbc9c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 34415 } } }
I20260924 23:18:25.754935 14720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [sys.catalog]: This master's current role is: LEADER
I20260924 23:18:25.754873 14719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "548b172f29eb4937b86835dbd7ebbc9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548b172f29eb4937b86835dbd7ebbc9c" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 34415 } } }
I20260924 23:18:25.755128 14719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c [sys.catalog]: This master's current role is: LEADER
I20260924 23:18:25.755165 14724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:18:25.755694 14724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:18:25.757505 14724 catalog_manager.cc:1347] Generated new cluster ID: d69c18ca407b4fd28fc74af47c80f02f
I20260924 23:18:25.757560 14724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:18:25 dist-test-slave-s1hw krb5kdc[14632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:25.764602 14724 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:18:25.765010 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:25.753697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57768 (local address 127.1.4.254:34415)
0924 23:18:25.753882 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:25.753893 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:25.753928 (+ 35us) server_negotiation.cc:407] Connection header received
0924 23:18:25.754057 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:25.754065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:25.754130 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:25.754267 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:25.754991 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:25.756455 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:25.757449 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:25.758030 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:25.758128 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:25.760885 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:25.760889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:25.760890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:25.760902 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:25.763428 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:25.763795 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:25.763798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:25.763798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:25.763920 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:25.764114 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:25.764114 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:25.764115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:25.764395 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:25.764551 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:25.764654 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 23:18:25.764794 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260924 23:18:25.765223 14724 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:18:25.765985 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:18:25.772234 14724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 548b172f29eb4937b86835dbd7ebbc9c: Generated new TSK 0
I20260924 23:18:25.772393 14724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:18:25.819800 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 14645
2026-09-24T23:18:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1257 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:18:25 dist-test-slave-s1hw krb5kdc[14740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:18:25 dist-test-slave-s1hw krb5kdc[14740](info): set up 2 sockets
Sep 24 23:18:25 dist-test-slave-s1hw krb5kdc[14740](info): commencing operation
krb5kdc: starting...
W20260924 23:18:26.771175 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:18:26 dist-test-slave-s1hw krb5kdc[14740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:18:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:18:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:18:26.888252 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38539
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:38539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:26.968190 14753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:26.968421 14753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:26.968451 14753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:26.969622 14753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:18:26.969666 14753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:26.969682 14753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:26.969698 14753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:18:26.969712 14753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:18:26.971220 14753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:38539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:26.971486 14753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:26.971688 14753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:26.976521 14761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:26.976511 14758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:26.976511 14759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:26.976735 14753 server_base.cc:1034] running on GCE node
I20260924 23:18:26.977018 14753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:26.977259 14753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:26.978399 14753 hybrid_clock.cc:648] HybridClock initialized: now 1790291906978381 us; error 30 us; skew 500 ppm
Sep 24 23:18:26 dist-test-slave-s1hw krb5kdc[14740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291906, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:26.980871 14753 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:18:26.981307 14753 webserver.cc:460] Webserver started at http://127.1.4.254:39367/ using document root <none> and password file <none>
I20260924 23:18:26.981508 14753 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:26.981550 14753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:26.981668 14753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:18:26.982759 14753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "d5a84caaa04d447f9f88cf67715793a3"
format_stamp: "Formatted at 2026-09-24 23:18:26 on dist-test-slave-s1hw"
server_key: "a82b652222d34e57bb56fe4e48e876eb"
server_key_iv: "c56d17398e90b559ece43978b4c386bd"
server_key_version: "encryptionkey@0"
I20260924 23:18:26.983104 14753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "d5a84caaa04d447f9f88cf67715793a3"
format_stamp: "Formatted at 2026-09-24 23:18:26 on dist-test-slave-s1hw"
server_key: "a82b652222d34e57bb56fe4e48e876eb"
server_key_iv: "c56d17398e90b559ece43978b4c386bd"
server_key_version: "encryptionkey@0"
I20260924 23:18:26.984331 14753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:18:26.984984 14768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:26.985128 14753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:18:26.985209 14753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "d5a84caaa04d447f9f88cf67715793a3"
format_stamp: "Formatted at 2026-09-24 23:18:26 on dist-test-slave-s1hw"
server_key: "a82b652222d34e57bb56fe4e48e876eb"
server_key_iv: "c56d17398e90b559ece43978b4c386bd"
server_key_version: "encryptionkey@0"
I20260924 23:18:26.985268 14753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:27.006248 14753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:27.008358 14753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:27.008499 14753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:27.012398 14753 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:38539
I20260924 23:18:27.012463 14820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:38539 every 8 connection(s)
I20260924 23:18:27.012786 14753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:18:27.012948 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 14753
I20260924 23:18:27.013055 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:18:27.013242 1043 external_mini_cluster.cc:1428] Setting key 82014f0808f9647d917cd46462c25cc1
I20260924 23:18:27.013913 14821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:27.016995 14821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3: Bootstrap starting.
I20260924 23:18:27.017580 14821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:27.017865 14821 log.cc:824] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:27.018555 14821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3: No bootstrap required, opened a new log
Sep 24 23:18:27 dist-test-slave-s1hw krb5kdc[14740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:27.020434 14821 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } }
I20260924 23:18:27.020613 14821 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:27.020653 14821 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5a84caaa04d447f9f88cf67715793a3, State: Initialized, Role: FOLLOWER
I20260924 23:18:27.020918 14821 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } }
I20260924 23:18:27.021109 14821 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:18:27.021176 14821 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:18:27.021234 14821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:27.021788 14821 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } }
I20260924 23:18:27.021879 14821 leader_election.cc:302] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5a84caaa04d447f9f88cf67715793a3; no voters:
I20260924 23:18:27.022042 14821 leader_election.cc:288] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:18:27.022084 14826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:18:27.022254 14826 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 1 LEADER]: Becoming Leader. State: Replica: d5a84caaa04d447f9f88cf67715793a3, State: Running, Role: LEADER
I20260924 23:18:27.022320 14821 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:18:27.022387 14826 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } }
I20260924 23:18:27.022800 14827 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5a84caaa04d447f9f88cf67715793a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } } }
I20260924 23:18:27.022881 14827 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [sys.catalog]: This master's current role is: LEADER
I20260924 23:18:27.023089 14828 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5a84caaa04d447f9f88cf67715793a3. Latest consensus state: current_term: 1 leader_uuid: "d5a84caaa04d447f9f88cf67715793a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } } }
I20260924 23:18:27.023415 14828 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [sys.catalog]: This master's current role is: LEADER
I20260924 23:18:27.023576 14835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:18:27.024122 14835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:18:27.024403 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.013802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.1.4.254:38539)
0924 23:18:27.014006 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:27.014016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:27.014049 (+ 33us) server_negotiation.cc:407] Connection header received
0924 23:18:27.014587 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:27.014598 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:27.014664 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:27.014834 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:27.015461 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.016796 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.017631 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.018205 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.018303 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.020729 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:27.020733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:27.020734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:27.020748 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:27.022479 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:27.022853 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:27.022856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:27.022857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:27.022957 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:27.023178 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:27.023180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:27.023180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:27.023570 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:27.023717 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:27.024039 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 23:18:27.024178 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260924 23:18:27.025877 14835 catalog_manager.cc:1347] Generated new cluster ID: 975588b2e67e42efab40d3531d855e07
I20260924 23:18:27.025928 14835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:18:27.032210 14835 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:18:27.032904 14835 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:18:27.037724 14835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3: Generated new TSK 0
I20260924 23:18:27.037905 14835 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:18:27.067920 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38539
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:27.147053 14847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:27.147260 14847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:27.147284 14847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:27.148483 14847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:27.148526 14847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:27.148553 14847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:18:27.150060 14847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:27.150313 14847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:27.150585 14847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:27.155583 14852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:27.155710 14847 server_base.cc:1034] running on GCE node
W20260924 23:18:27.155609 14855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:27.155583 14853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:27.156064 14847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:27.156311 14847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:27.157469 14847 hybrid_clock.cc:648] HybridClock initialized: now 1790291907157446 us; error 41 us; skew 500 ppm
Sep 24 23:18:27 dist-test-slave-s1hw krb5kdc[14740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:27.159969 14847 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:18:27.160406 14847 webserver.cc:460] Webserver started at http://127.1.4.193:41847/ using document root <none> and password file <none>
I20260924 23:18:27.160625 14847 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:27.160670 14847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:27.160789 14847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:18:27.161800 14847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d"
format_stamp: "Formatted at 2026-09-24 23:18:27 on dist-test-slave-s1hw"
server_key: "5eea3dae2bc7398cf7147c93e64fe921"
server_key_iv: "413335f1a701b9f55440039e2d76a63a"
server_key_version: "encryptionkey@0"
I20260924 23:18:27.162132 14847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d"
format_stamp: "Formatted at 2026-09-24 23:18:27 on dist-test-slave-s1hw"
server_key: "5eea3dae2bc7398cf7147c93e64fe921"
server_key_iv: "413335f1a701b9f55440039e2d76a63a"
server_key_version: "encryptionkey@0"
I20260924 23:18:27.163339 14847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 23:18:27.164093 14862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:27.164242 14847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:18:27.164322 14847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d"
format_stamp: "Formatted at 2026-09-24 23:18:27 on dist-test-slave-s1hw"
server_key: "5eea3dae2bc7398cf7147c93e64fe921"
server_key_iv: "413335f1a701b9f55440039e2d76a63a"
server_key_version: "encryptionkey@0"
I20260924 23:18:27.164383 14847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:27.186724 14847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:27.189016 14847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:27.189172 14847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:27.189404 14847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:27.189723 14847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:18:27.189759 14847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:27.189792 14847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:18:27.189819 14847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:27.196084 14847 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:38711
I20260924 23:18:27.196133 14975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:38711 every 8 connection(s)
I20260924 23:18:27.196478 14847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:18:27 dist-test-slave-s1hw krb5kdc[14740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:27.202662 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 14847
I20260924 23:18:27.202795 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:18:27.203037 1043 external_mini_cluster.cc:1428] Setting key 74c0178401ed13a6dd3e56b9cc65c30b
I20260924 23:18:27.206101 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.197229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:40711 (local address 127.1.4.254:38539)
0924 23:18:27.197301 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:27.197303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:27.197490 (+ 187us) server_negotiation.cc:407] Connection header received
0924 23:18:27.198019 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:27.198021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:27.198032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:27.198088 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:27.199024 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.199584 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.200639 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.200836 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.200884 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.203199 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:27.203201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:27.203202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:27.203209 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:27.204639 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:27.205119 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:27.205121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:27.205121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:27.205187 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:27.205418 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:27.205419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:27.205419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:27.205549 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:27.205609 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:27.205917 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 23:18:27.206010 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 23:18:27.206183 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.197184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:40711)
0924 23:18:27.197375 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:27.197414 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:27.197908 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:27.198177 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:27.198181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:27.198748 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:27.198952 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:27.198957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.199686 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.199687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:27.200555 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:27.200558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.200910 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.200911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:27.201018 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.201466 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:27.201475 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:27.203075 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:27.204720 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:27.204729 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:27.204731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:27.205051 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:27.205251 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:27.205252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:27.205252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:27.205363 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:27.205602 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:27.205605 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:27.205727 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 23:18:27.205880 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 23:18:27.206011 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260924 23:18:27.206626 14976 heartbeater.cc:347] Connected to a master server at 127.1.4.254:38539
I20260924 23:18:27.206713 14976 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:27.206889 14976 heartbeater.cc:511] Master 127.1.4.254:38539 requested a full tablet report, sending...
I20260924 23:18:27.207486 14785 ts_manager.cc:194] Registered new tserver with Master: 9ebcbc07144b4cb3bf5d68848fa4e24d (127.1.4.193:38711)
I20260924 23:18:27.208410 14785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:40711
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:18:27.230279 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38539
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:27.309211 14981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:27.309433 14981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:27.309458 14981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:27.310681 14981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:27.310732 14981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:27.310757 14981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:18:27.312237 14981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:27.312463 14981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:27.312682 14981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:27.317603 14987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:27.317669 14986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:27.317610 14989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:27.318135 14981 server_base.cc:1034] running on GCE node
I20260924 23:18:27.318434 14981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:27.318682 14981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:27.319928 14981 hybrid_clock.cc:648] HybridClock initialized: now 1790291907319891 us; error 51 us; skew 500 ppm
Sep 24 23:18:27 dist-test-slave-s1hw krb5kdc[14740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:27.322695 14981 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:18:27.323186 14981 webserver.cc:460] Webserver started at http://127.1.4.194:45435/ using document root <none> and password file <none>
I20260924 23:18:27.323493 14981 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:27.323544 14981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:27.323678 14981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:18:27.324779 14981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "2c19ed3773504dba845328975d91e1f2"
format_stamp: "Formatted at 2026-09-24 23:18:27 on dist-test-slave-s1hw"
server_key: "ebc1cd33dd37c6f70d3f50f39076d21c"
server_key_iv: "f9b1fe9265d0ebf04c999d31828d59cc"
server_key_version: "encryptionkey@0"
I20260924 23:18:27.325111 14981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "2c19ed3773504dba845328975d91e1f2"
format_stamp: "Formatted at 2026-09-24 23:18:27 on dist-test-slave-s1hw"
server_key: "ebc1cd33dd37c6f70d3f50f39076d21c"
server_key_iv: "f9b1fe9265d0ebf04c999d31828d59cc"
server_key_version: "encryptionkey@0"
I20260924 23:18:27.326366 14981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:18:27.327068 14996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:27.327232 14981 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:18:27.327313 14981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "2c19ed3773504dba845328975d91e1f2"
format_stamp: "Formatted at 2026-09-24 23:18:27 on dist-test-slave-s1hw"
server_key: "ebc1cd33dd37c6f70d3f50f39076d21c"
server_key_iv: "f9b1fe9265d0ebf04c999d31828d59cc"
server_key_version: "encryptionkey@0"
I20260924 23:18:27.327414 14981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:27.340574 14981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:27.342449 14981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:27.342590 14981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:27.342804 14981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:27.343097 14981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:18:27.343129 14981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:27.343159 14981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:18:27.343176 14981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:27.348655 14981 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:33269
I20260924 23:18:27.348747 15109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:33269 every 8 connection(s)
I20260924 23:18:27.349035 14981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:18:27.354887 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 14981
I20260924 23:18:27.355006 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
Sep 24 23:18:27 dist-test-slave-s1hw krb5kdc[14740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:27.355185 1043 external_mini_cluster.cc:1428] Setting key c1ebe719f71decdd27157ad9ba5cf836
I20260924 23:18:27.358438 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.349709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:45191 (local address 127.1.4.254:38539)
0924 23:18:27.349795 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:27.349797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:27.350118 (+ 321us) server_negotiation.cc:407] Connection header received
0924 23:18:27.350544 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:27.350545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:27.350556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:27.350607 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:27.351572 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.352099 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.353071 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.353282 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.353340 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.355618 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:27.355620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:27.355620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:27.355629 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:27.356962 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:27.357390 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:27.357391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:27.357391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:27.357447 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:27.357686 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:27.357687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:27.357687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:27.357884 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:27.357954 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:27.358256 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 23:18:27.358349 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 23:18:27.358541 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.349686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:45191)
0924 23:18:27.349914 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:27.349984 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:27.350465 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:27.350670 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:27.350673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:27.351231 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:27.351476 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:27.351482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.352169 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.352170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:27.352984 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:27.352986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.353345 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.353346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:27.353448 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.354024 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:27.354032 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:27.355534 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:27.357036 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:27.357045 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:27.357047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:27.357317 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:27.357502 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:27.357502 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:27.357503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:27.357626 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:27.357968 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:27.357974 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:27.358127 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 23:18:27.358220 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 23:18:27.358352 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":156,"threads_started":1}
I20260924 23:18:27.358989 15110 heartbeater.cc:347] Connected to a master server at 127.1.4.254:38539
I20260924 23:18:27.359087 15110 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:27.359289 15110 heartbeater.cc:511] Master 127.1.4.254:38539 requested a full tablet report, sending...
I20260924 23:18:27.359711 14785 ts_manager.cc:194] Registered new tserver with Master: 2c19ed3773504dba845328975d91e1f2 (127.1.4.194:33269)
I20260924 23:18:27.360239 14785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:45191
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:18:27.382452 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38539
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:27.463222 15115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:27.463469 15115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:27.463500 15115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:27.464656 15115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:27.464702 15115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:27.464737 15115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:18:27.466228 15115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:27.466466 15115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:27.466679 15115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:18:27.471844 15115 server_base.cc:1034] running on GCE node
W20260924 23:18:27.471819 15121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:27.471851 15120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:27.471818 15123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:27.472256 15115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:27.472465 15115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:27.473591 15115 hybrid_clock.cc:648] HybridClock initialized: now 1790291907473582 us; error 27 us; skew 500 ppm
Sep 24 23:18:27 dist-test-slave-s1hw krb5kdc[14740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:27.476049 15115 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:18:27.476516 15115 webserver.cc:460] Webserver started at http://127.1.4.195:41039/ using document root <none> and password file <none>
I20260924 23:18:27.476716 15115 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:27.476763 15115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:27.476897 15115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:18:27.477923 15115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "4045123d7a14416eb00f4978335ac53e"
format_stamp: "Formatted at 2026-09-24 23:18:27 on dist-test-slave-s1hw"
server_key: "15928bd7c6b248ae8fd97b6c11672e73"
server_key_iv: "82996ed268404362857123fa18e38d62"
server_key_version: "encryptionkey@0"
I20260924 23:18:27.478250 15115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "4045123d7a14416eb00f4978335ac53e"
format_stamp: "Formatted at 2026-09-24 23:18:27 on dist-test-slave-s1hw"
server_key: "15928bd7c6b248ae8fd97b6c11672e73"
server_key_iv: "82996ed268404362857123fa18e38d62"
server_key_version: "encryptionkey@0"
I20260924 23:18:27.479521 15115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:18:27.480298 15130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:27.480513 15115 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:27.480597 15115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "4045123d7a14416eb00f4978335ac53e"
format_stamp: "Formatted at 2026-09-24 23:18:27 on dist-test-slave-s1hw"
server_key: "15928bd7c6b248ae8fd97b6c11672e73"
server_key_iv: "82996ed268404362857123fa18e38d62"
server_key_version: "encryptionkey@0"
I20260924 23:18:27.480661 15115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:27.489745 15115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:27.492178 15115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:27.492295 15115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:27.492463 15115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:27.492753 15115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:18:27.492785 15115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:27.492816 15115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:18:27.492838 15115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:27.498265 15115 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:46597
I20260924 23:18:27.498333 15243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:46597 every 8 connection(s)
I20260924 23:18:27.498631 15115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
Sep 24 23:18:27 dist-test-slave-s1hw krb5kdc[14740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:27.507099 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 15115
I20260924 23:18:27.507201 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:18:27.507382 1043 external_mini_cluster.cc:1428] Setting key 3fb8a1fdec986284a5f351463b4d0459
I20260924 23:18:27.507448 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.499351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:55553 (local address 127.1.4.254:38539)
0924 23:18:27.499441 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:27.499442 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:27.499675 (+ 233us) server_negotiation.cc:407] Connection header received
0924 23:18:27.500174 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:27.500175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:27.500187 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:27.500244 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:27.501156 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.501654 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.502632 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.502832 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.502894 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.505003 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:27.505005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:27.505006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:27.505013 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:27.506145 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:27.506525 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:27.506527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:27.506527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:27.506578 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:27.506794 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:27.506795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:27.506795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:27.506930 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:27.506987 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:27.507243 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 23:18:27.507325 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 23:18:27.507624 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.499325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:55553)
0924 23:18:27.499546 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:27.499583 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:27.500080 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:27.500297 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:27.500301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:27.500873 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:27.501086 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:27.501091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.501726 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.501726 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:27.502529 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:27.502532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.502903 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.502904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:27.503016 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.503532 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:27.503542 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:27.504915 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:27.506224 (+ 1309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:27.506232 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:27.506234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:27.506458 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:27.506635 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:27.506635 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:27.506636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:27.506739 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:27.506989 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:27.506995 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:27.507131 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 23:18:27.507219 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 23:18:27.507403 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":152,"threads_started":1}
I20260924 23:18:27.508106 15244 heartbeater.cc:347] Connected to a master server at 127.1.4.254:38539
I20260924 23:18:27.508205 15244 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:27.508381 15244 heartbeater.cc:511] Master 127.1.4.254:38539 requested a full tablet report, sending...
I20260924 23:18:27.508687 14785 ts_manager.cc:194] Registered new tserver with Master: 4045123d7a14416eb00f4978335ac53e (127.1.4.195:46597)
I20260924 23:18:27.509075 14785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:55553
I20260924 23:18:27.519172 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:18:27.526383 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.520551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32870 (local address 127.1.4.254:38539)
0924 23:18:27.520604 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:27.520606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:27.520665 (+ 59us) server_negotiation.cc:407] Connection header received
0924 23:18:27.520725 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:27.520725 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:27.520734 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:27.520787 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:27.521345 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.521900 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.522688 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.522953 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.523008 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.524291 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:27.524293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:27.524294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:27.524301 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:27.525488 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:27.525773 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:27.525773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:27.525774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:27.525826 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:27.525977 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:27.525977 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:27.525978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:27.526111 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:27.526179 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:27.526257 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 23:18:27.526316 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 23:18:27.527922 14785 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:18:27.529529 14785 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:18:27.536037 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.532100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47082 (local address 127.1.4.195:46597)
0924 23:18:27.532255 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:27.532260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:27.532269 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:18:27.532305 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:27.532306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:27.532321 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:27.532388 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:27.532591 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.533742 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.534791 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.535561 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.535633 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.535697 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:27.535888 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 23:18:27.535938 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260924 23:18:27.536057 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.531722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:38711 (local address 127.0.0.1:54432)
0924 23:18:27.531986 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:27.532029 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:27.532098 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:27.532371 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:27.532374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:27.532377 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:27.532456 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:27.532458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.533838 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.533838 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:27.535002 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:27.535004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.535749 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.535750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:27.535811 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.535815 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:18:27.535870 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 23:18:27.535956 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":52,"thread_start_us":56,"threads_started":1}
I20260924 23:18:27.536180 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.531833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48726 (local address 127.1.4.194:33269)
0924 23:18:27.532013 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:27.532019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:27.532264 (+ 245us) server_negotiation.cc:407] Connection header received
0924 23:18:27.532305 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:27.532307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:27.532321 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:27.532385 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:27.532615 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.533471 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.534987 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.535719 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.535777 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.535831 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:27.536056 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 23:18:27.536098 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
I20260924 23:18:27.536384 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.531989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.1.4.193:38711)
0924 23:18:27.532189 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:27.532193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:27.532213 (+ 20us) server_negotiation.cc:407] Connection header received
0924 23:18:27.532253 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:27.532255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:27.532272 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:27.532362 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:27.532542 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.533736 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.535074 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.535641 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.535740 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.535805 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:27.536264 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 23:18:27.536309 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":98,"threads_started":1}
I20260924 23:18:27.536656 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.531723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:46597 (local address 127.0.0.1:47082)
0924 23:18:27.532182 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:27.532191 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:27.532226 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:27.532477 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:27.532477 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:27.532479 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:27.532521 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:27.532523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.533846 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.533847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:27.534729 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:27.534732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.535686 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.535687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:27.535784 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.535799 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 23:18:27.535875 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 23:18:27.535964 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":47,"thread_start_us":54,"threads_started":1}
I20260924 23:18:27.536727 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.531725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:33269 (local address 127.0.0.1:48726)
0924 23:18:27.531986 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:27.532029 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:27.532098 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:27.532476 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:27.532478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:27.532483 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:27.532542 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:27.532545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.533573 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.533574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:27.534897 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:27.534900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.535860 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.535861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:27.535914 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.535920 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 23:18:27.535960 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 23:18:27.535996 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":34,"spinlock_wait_cycles":384,"thread_start_us":66,"threads_started":1}
I20260924 23:18:27.537168 14910 tablet_service.cc:1467] Processing CreateTablet for tablet be1e2f4180cc43a793b5e25c00301555 (DEFAULT_TABLE table=test-table [id=915d785cc08144b281cb907295688e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:18:27.537334 15044 tablet_service.cc:1467] Processing CreateTablet for tablet be1e2f4180cc43a793b5e25c00301555 (DEFAULT_TABLE table=test-table [id=915d785cc08144b281cb907295688e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:18:27.537348 15178 tablet_service.cc:1467] Processing CreateTablet for tablet be1e2f4180cc43a793b5e25c00301555 (DEFAULT_TABLE table=test-table [id=915d785cc08144b281cb907295688e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:18:27.537541 14910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1e2f4180cc43a793b5e25c00301555. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:27.537614 15044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1e2f4180cc43a793b5e25c00301555. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:27.537672 15178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1e2f4180cc43a793b5e25c00301555. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:27.540464 15258 tablet_bootstrap.cc:492] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e: Bootstrap starting.
I20260924 23:18:27.540807 15259 tablet_bootstrap.cc:492] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2: Bootstrap starting.
I20260924 23:18:27.541342 15258 tablet_bootstrap.cc:654] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:27.541401 15259 tablet_bootstrap.cc:654] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:27.541898 15259 log.cc:824] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:27.541898 15258 log.cc:824] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:27.542227 15260 tablet_bootstrap.cc:492] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Bootstrap starting.
I20260924 23:18:27.542670 15259 tablet_bootstrap.cc:492] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2: No bootstrap required, opened a new log
I20260924 23:18:27.542678 15258 tablet_bootstrap.cc:492] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e: No bootstrap required, opened a new log
I20260924 23:18:27.542745 15259 ts_tablet_manager.cc:1397] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:18:27.542745 15258 ts_tablet_manager.cc:1397] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:18:27.542856 15260 tablet_bootstrap.cc:654] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:27.543295 15260 log.cc:824] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:27.543938 15260 tablet_bootstrap.cc:492] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: No bootstrap required, opened a new log
I20260924 23:18:27.543998 15260 ts_tablet_manager.cc:1397] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:18:27.544648 15258 raft_consensus.cc:348] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:27.544648 15259 raft_consensus.cc:348] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:27.544838 15258 raft_consensus.cc:374] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:27.544838 15259 raft_consensus.cc:374] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:27.544873 15259 raft_consensus.cc:729] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c19ed3773504dba845328975d91e1f2, State: Initialized, Role: FOLLOWER
I20260924 23:18:27.544873 15258 raft_consensus.cc:729] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4045123d7a14416eb00f4978335ac53e, State: Initialized, Role: FOLLOWER
I20260924 23:18:27.544999 15258 consensus_queue.cc:260] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:27.545022 15259 consensus_queue.cc:260] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:27.545281 15258 ts_tablet_manager.cc:1428] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:18:27.545281 15259 ts_tablet_manager.cc:1428] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:18:27.545394 15244 heartbeater.cc:503] Master 127.1.4.254:38539 was elected leader, sending a full tablet report...
I20260924 23:18:27.545436 15110 heartbeater.cc:503] Master 127.1.4.254:38539 was elected leader, sending a full tablet report...
I20260924 23:18:27.545569 15260 raft_consensus.cc:348] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:27.545740 15260 raft_consensus.cc:374] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:27.545780 15260 raft_consensus.cc:729] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebcbc07144b4cb3bf5d68848fa4e24d, State: Initialized, Role: FOLLOWER
I20260924 23:18:27.545903 15260 consensus_queue.cc:260] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:27.546164 15260 ts_tablet_manager.cc:1428] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:18:27.546182 14976 heartbeater.cc:503] Master 127.1.4.254:38539 was elected leader, sending a full tablet report...
W20260924 23:18:27.599627 15111 tablet.cc:2369] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:18:27.697213 14977 tablet.cc:2369] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:18:27.749162 15245 tablet.cc:2369] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:18:27.771942 15264 raft_consensus.cc:482] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:18:27.772069 15264 raft_consensus.cc:504] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:27.772508 15264 leader_election.cc:288] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c19ed3773504dba845328975d91e1f2 (127.1.4.194:33269), 9ebcbc07144b4cb3bf5d68848fa4e24d (127.1.4.193:38711)
I20260924 23:18:27.776566 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.772784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:36621 (local address 127.1.4.194:33269)
0924 23:18:27.772827 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:27.772829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:27.773050 (+ 221us) server_negotiation.cc:407] Connection header received
0924 23:18:27.773078 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:27.773079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:27.773091 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:27.773134 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:27.773272 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.774003 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.775633 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.776200 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.776272 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.776284 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:27.776445 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 23:18:27.776489 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 23:18:27.776566 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.772869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:40755 (local address 127.1.4.193:38711)
0924 23:18:27.772937 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:27.772939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:27.772950 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:18:27.772981 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:27.772982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:27.772996 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:27.773057 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:27.773222 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.773984 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.775633 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.776199 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.776272 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.776284 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:27.776460 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 23:18:27.776494 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 23:18:27.776580 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.772711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:38711 (local address 127.1.4.195:40755)
0924 23:18:27.772761 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:27.772774 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:27.772842 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:27.773076 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:27.773077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:27.773078 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:27.773148 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:27.773150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.774106 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.774107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:27.775527 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:27.775531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.776307 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.776308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:27.776370 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.776374 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:18:27.776434 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 23:18:27.776482 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":27}
I20260924 23:18:27.776587 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.772711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:33269 (local address 127.1.4.195:36621)
0924 23:18:27.772966 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:27.772974 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:27.773019 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:27.773148 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:27.773149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:27.773150 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:27.773207 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:27.773208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.774106 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.774107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:27.775526 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:27.775531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.776290 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.776291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:27.776370 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.776375 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:18:27.776434 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 23:18:27.776482 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":157,"thread_start_us":139,"threads_started":1}
I20260924 23:18:27.776947 14930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1e2f4180cc43a793b5e25c00301555" candidate_uuid: "4045123d7a14416eb00f4978335ac53e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" is_pre_election: true
I20260924 23:18:27.776985 15064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1e2f4180cc43a793b5e25c00301555" candidate_uuid: "4045123d7a14416eb00f4978335ac53e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c19ed3773504dba845328975d91e1f2" is_pre_election: true
I20260924 23:18:27.777117 14930 raft_consensus.cc:2454] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4045123d7a14416eb00f4978335ac53e in term 0.
I20260924 23:18:27.777124 15064 raft_consensus.cc:2454] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4045123d7a14416eb00f4978335ac53e in term 0.
I20260924 23:18:27.777339 15133 leader_election.cc:302] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c19ed3773504dba845328975d91e1f2, 4045123d7a14416eb00f4978335ac53e; no voters:
I20260924 23:18:27.777536 15264 raft_consensus.cc:2781] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:18:27.777580 15264 raft_consensus.cc:482] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:18:27.777601 15264 raft_consensus.cc:3037] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:27.778280 15264 raft_consensus.cc:504] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:27.778415 15264 leader_election.cc:288] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [CANDIDATE]: Term 1 election: Requested vote from peers 2c19ed3773504dba845328975d91e1f2 (127.1.4.194:33269), 9ebcbc07144b4cb3bf5d68848fa4e24d (127.1.4.193:38711)
I20260924 23:18:27.778601 14930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1e2f4180cc43a793b5e25c00301555" candidate_uuid: "4045123d7a14416eb00f4978335ac53e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d"
I20260924 23:18:27.778620 15064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1e2f4180cc43a793b5e25c00301555" candidate_uuid: "4045123d7a14416eb00f4978335ac53e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c19ed3773504dba845328975d91e1f2"
I20260924 23:18:27.778678 14930 raft_consensus.cc:3037] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:27.778678 15064 raft_consensus.cc:3037] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:27.779337 15064 raft_consensus.cc:2454] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4045123d7a14416eb00f4978335ac53e in term 1.
I20260924 23:18:27.779506 14930 raft_consensus.cc:2454] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4045123d7a14416eb00f4978335ac53e in term 1.
I20260924 23:18:27.779561 15133 leader_election.cc:302] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c19ed3773504dba845328975d91e1f2, 4045123d7a14416eb00f4978335ac53e; no voters:
I20260924 23:18:27.779651 15264 raft_consensus.cc:2781] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:18:27.779801 15264 raft_consensus.cc:686] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [term 1 LEADER]: Becoming Leader. State: Replica: 4045123d7a14416eb00f4978335ac53e, State: Running, Role: LEADER
I20260924 23:18:27.779891 15264 consensus_queue.cc:237] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:27.780694 14783 catalog_manager.cc:5495] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4045123d7a14416eb00f4978335ac53e (127.1.4.195). New cstate: current_term: 1 leader_uuid: "4045123d7a14416eb00f4978335ac53e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260924 23:18:27.821484 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.819076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.1.4.195:46597)
0924 23:18:27.819147 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:27.819148 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:27.819157 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:18:27.819195 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:27.819196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:27.819217 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:27.819284 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:18:27.819454 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.820028 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.820793 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:27.820986 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:27.821048 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:27.821160 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:18:27.821271 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:18:27.821328 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:27.821361 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 23:18:27.821379 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 23:18:27.823526 14930 raft_consensus.cc:1261] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Refusing update from remote peer 4045123d7a14416eb00f4978335ac53e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:18:27.823527 15064 raft_consensus.cc:1261] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2 [term 1 FOLLOWER]: Refusing update from remote peer 4045123d7a14416eb00f4978335ac53e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:18:27.823899 15268 consensus_queue.cc:1035] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:18:27.824045 15264 consensus_queue.cc:1035] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:18:27.826224 15271 mvcc.cc:205] Tried to move back new op lower bound from 7333035654441762816 to 7333035654266974208. Current Snapshot: MvccSnapshot[applied={T|T < 7333035654441762816}]
I20260924 23:18:27.826579 15272 mvcc.cc:205] Tried to move back new op lower bound from 7333035654441762816 to 7333035654266974208. Current Snapshot: MvccSnapshot[applied={T|T < 7333035654441762816}]
I20260924 23:18:27.826658 15273 mvcc.cc:205] Tried to move back new op lower bound from 7333035654441762816 to 7333035654266974208. Current Snapshot: MvccSnapshot[applied={T|T < 7333035654441762816}]
I20260924 23:18:27.829056 14783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:18:27.829164 14783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:18:27.830014 14783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3: Sending DeleteTablet for 3 replicas of tablet be1e2f4180cc43a793b5e25c00301555
I20260924 23:18:27.830344 15044 tablet_service.cc:1514] Processing DeleteTablet for tablet be1e2f4180cc43a793b5e25c00301555 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:18:27 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:48726
I20260924 23:18:27.830377 15178 tablet_service.cc:1514] Processing DeleteTablet for tablet be1e2f4180cc43a793b5e25c00301555 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:18:27 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:47082
I20260924 23:18:27.830601 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 14847
I20260924 23:18:27.830614 15280 tablet_replica.cc:330] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2: stopping tablet replica
I20260924 23:18:27.830747 15280 raft_consensus.cc:2229] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:18:27.830865 15280 raft_consensus.cc:2258] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:18:27.831032 15281 tablet_replica.cc:330] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e: stopping tablet replica
I20260924 23:18:27.831110 15281 raft_consensus.cc:2229] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:18:27.831228 15281 raft_consensus.cc:2258] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:18:27.831710 15281 ts_tablet_manager.cc:1905] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:18:27.833415 15280 ts_tablet_manager.cc:1905] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:18:27.833515 15281 ts_tablet_manager.cc:1918] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:18:27.833591 15281 log.cc:1196] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/be1e2f4180cc43a793b5e25c00301555
I20260924 23:18:27.833889 15281 ts_tablet_manager.cc:1939] T be1e2f4180cc43a793b5e25c00301555 P 4045123d7a14416eb00f4978335ac53e: Deleting consensus metadata
I20260924 23:18:27.834576 14770 catalog_manager.cc:4841] TS 4045123d7a14416eb00f4978335ac53e (127.1.4.195:46597): tablet be1e2f4180cc43a793b5e25c00301555 (table test-table [id=915d785cc08144b281cb907295688e46]) successfully deleted
I20260924 23:18:27.835388 15280 ts_tablet_manager.cc:1918] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:18:27.835520 15280 log.cc:1196] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/be1e2f4180cc43a793b5e25c00301555
I20260924 23:18:27.835804 15280 ts_tablet_manager.cc:1939] T be1e2f4180cc43a793b5e25c00301555 P 2c19ed3773504dba845328975d91e1f2: Deleting consensus metadata
I20260924 23:18:27.836273 14771 catalog_manager.cc:4841] TS 2c19ed3773504dba845328975d91e1f2 (127.1.4.194:33269): tablet be1e2f4180cc43a793b5e25c00301555 (table test-table [id=915d785cc08144b281cb907295688e46]) successfully deleted
W20260924 23:18:27.836951 14769 connection.cc:535] client connection to 127.1.4.193:38711 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 23:18:27.837057 14769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 23:18:27.837123 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 14981
I20260924 23:18:27.838060 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:27.837361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:38711 (local address 127.0.0.1:54444)
0924 23:18:27.837991 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:27.838030 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:38711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":611}
W20260924 23:18:27.839251 14769 catalog_manager.cc:4584] TS 9ebcbc07144b4cb3bf5d68848fa4e24d (127.1.4.193:38711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be1e2f4180cc43a793b5e25c00301555: Network error: Client connection negotiation failed: client connection to 127.1.4.193:38711: connect: Connection refused (error 111)
I20260924 23:18:27.842089 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 15115
I20260924 23:18:27.846199 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 14753
I20260924 23:18:27.850543 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38539
--webserver_interface=127.1.4.254
--webserver_port=39367
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:38539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:27.930725 15283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:27.930938 15283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:27.930966 15283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:27.932140 15283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:18:27.932181 15283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:27.932199 15283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:27.932214 15283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:18:27.932229 15283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:18:27.933658 15283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:38539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=39367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:27.933905 15283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:27.934108 15283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:27.942698 15290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:27.942823 15283 server_base.cc:1034] running on GCE node
W20260924 23:18:27.942703 15289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:27.942703 15292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:27.943158 15283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:27.943511 15283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:27.944674 15283 hybrid_clock.cc:648] HybridClock initialized: now 1790291907944657 us; error 27 us; skew 500 ppm
Sep 24 23:18:27 dist-test-slave-s1hw krb5kdc[14740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291907, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:27.952979 15283 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:18:27.953536 15283 webserver.cc:460] Webserver started at http://127.1.4.254:39367/ using document root <none> and password file <none>
I20260924 23:18:27.953783 15283 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:27.953852 15283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:27.955349 15283 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:27.956247 15299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:27.956444 15283 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:18:27.956542 15283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "d5a84caaa04d447f9f88cf67715793a3"
format_stamp: "Formatted at 2026-09-24 23:18:26 on dist-test-slave-s1hw"
server_key: "a82b652222d34e57bb56fe4e48e876eb"
server_key_iv: "c56d17398e90b559ece43978b4c386bd"
server_key_version: "encryptionkey@0"
I20260924 23:18:27.957005 15283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:27.988370 15283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:27.990316 15283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:27.990468 15283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:27.994287 15283 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:38539
I20260924 23:18:27.994324 15351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:38539 every 8 connection(s)
I20260924 23:18:27.994685 15283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:18:27.995838 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 15283
I20260924 23:18:27.996088 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:38711
--local_ip_for_outbound_sockets=127.1.4.193
--tserver_master_addrs=127.1.4.254:38539
--webserver_port=41847
--webserver_interface=127.1.4.193
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:18:27.996374 15352 sys_catalog.cc:263] Verifying existing consensus state
I20260924 23:18:27.997046 15352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3: Bootstrap starting.
I20260924 23:18:28.005009 15352 log.cc:824] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:28.006974 15352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:18:28.007248 15352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3: Bootstrap complete.
I20260924 23:18:28.009443 15352 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } }
I20260924 23:18:28.009685 15352 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5a84caaa04d447f9f88cf67715793a3, State: Initialized, Role: FOLLOWER
I20260924 23:18:28.009842 15352 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } }
I20260924 23:18:28.009915 15352 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:18:28.009972 15352 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:18:28.010021 15352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 1 FOLLOWER]: Advancing to term 2
I20260924 23:18:28.011109 15352 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } }
I20260924 23:18:28.011214 15352 leader_election.cc:302] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5a84caaa04d447f9f88cf67715793a3; no voters:
I20260924 23:18:28.011427 15352 leader_election.cc:288] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 23:18:28.011456 15356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 23:18:28.011560 15356 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 2 LEADER]: Becoming Leader. State: Replica: d5a84caaa04d447f9f88cf67715793a3, State: Running, Role: LEADER
I20260924 23:18:28.011657 15356 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } }
I20260924 23:18:28.011709 15352 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:18:28.011955 15357 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d5a84caaa04d447f9f88cf67715793a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } } }
I20260924 23:18:28.012053 15357 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [sys.catalog]: This master's current role is: LEADER
I20260924 23:18:28.012161 15356 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5a84caaa04d447f9f88cf67715793a3. Latest consensus state: current_term: 2 leader_uuid: "d5a84caaa04d447f9f88cf67715793a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } } }
I20260924 23:18:28.012233 15356 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [sys.catalog]: This master's current role is: LEADER
I20260924 23:18:28.012586 15361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:18:28.013410 15361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:18:28.013535 15361 catalog_manager.cc:1259] Loaded cluster ID: 975588b2e67e42efab40d3531d855e07
I20260924 23:18:28.013765 15361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:18:28.014006 15361 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 23:18:28.016727 15361 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 23:18:28.024889 15361 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:18:28.025144 15361 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3: Loaded TSK: 0
I20260924 23:18:28.025365 15361 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 23:18:28.092656 15354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:28.092872 15354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:28.092905 15354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:28.094108 15354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:28.094156 15354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:28.094190 15354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:18:28.095724 15354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:38711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=41847
--tserver_master_addrs=127.1.4.254:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:28.095980 15354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:28.096189 15354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:28.101349 15382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:28.101435 15379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:28.101349 15380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:28.101739 15354 server_base.cc:1034] running on GCE node
I20260924 23:18:28.102039 15354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:28.102346 15354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:28.103538 15354 hybrid_clock.cc:648] HybridClock initialized: now 1790291908103524 us; error 39 us; skew 500 ppm
Sep 24 23:18:28 dist-test-slave-s1hw krb5kdc[14740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291908, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:28.106485 15354 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:18:28.106927 15354 webserver.cc:460] Webserver started at http://127.1.4.193:41847/ using document root <none> and password file <none>
I20260924 23:18:28.107148 15354 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:28.107209 15354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:28.108618 15354 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:28.109311 15389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:28.109472 15354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:28.109553 15354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d"
format_stamp: "Formatted at 2026-09-24 23:18:27 on dist-test-slave-s1hw"
server_key: "5eea3dae2bc7398cf7147c93e64fe921"
server_key_iv: "413335f1a701b9f55440039e2d76a63a"
server_key_version: "encryptionkey@0"
I20260924 23:18:28.109901 15354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:28.121587 15354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:28.123821 15354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:28.123965 15354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:28.124182 15354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:28.124606 15396 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 23:18:28.126559 15354 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 23:18:28.126605 15354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 23:18:28.126637 15354 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 23:18:28.127241 15354 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 23:18:28.127274 15354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 23:18:28.127408 15396 tablet_bootstrap.cc:492] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Bootstrap starting.
I20260924 23:18:28.132793 15354 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:38711
I20260924 23:18:28.133250 15354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:18:28.133833 15503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:38711 every 8 connection(s)
I20260924 23:18:28.138095 15396 log.cc:824] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:28.140894 15396 tablet_bootstrap.cc:492] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
Sep 24 23:18:28 dist-test-slave-s1hw krb5kdc[14740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291908, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:28.141207 15396 tablet_bootstrap.cc:492] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Bootstrap complete.
I20260924 23:18:28.142315 15396 ts_tablet_manager.cc:1397] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260924 23:18:28.142679 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 15354
I20260924 23:18:28.142961 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:33269
--local_ip_for_outbound_sockets=127.1.4.194
--tserver_master_addrs=127.1.4.254:38539
--webserver_port=45435
--webserver_interface=127.1.4.194
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:18:28.143702 15396 raft_consensus.cc:348] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:28.144050 15396 raft_consensus.cc:729] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebcbc07144b4cb3bf5d68848fa4e24d, State: Initialized, Role: FOLLOWER
I20260924 23:18:28.144225 15396 consensus_queue.cc:260] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:28.144527 15396 ts_tablet_manager.cc:1428] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260924 23:18:28.146175 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.134586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:59789)
0924 23:18:28.134871 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:28.134907 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:28.135353 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:28.135887 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:28.135892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:28.136248 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:28.136454 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:28.136459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.137686 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.137687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:28.138631 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:28.138634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.139319 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.139320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:28.139586 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.140062 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:28.140070 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:28.141467 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:28.144283 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:28.144291 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:28.144293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:28.144591 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:28.144891 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:28.144892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:28.144892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:28.145028 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:28.145539 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:28.145545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:28.145707 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 23:18:28.145813 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 23:18:28.145981 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":93,"threads_started":1}
I20260924 23:18:28.146262 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.134689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:59789 (local address 127.1.4.254:38539)
0924 23:18:28.135529 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:28.135539 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:28.135568 (+ 29us) server_negotiation.cc:407] Connection header received
0924 23:18:28.135677 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:28.135681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:28.135725 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:28.135839 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:28.136574 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.137578 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.138712 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.139211 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.139311 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.141564 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:28.141568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:28.141569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:28.141582 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:28.143415 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:28.144691 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:28.144695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:28.144695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:28.144820 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:28.145119 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:28.145121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:28.145121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:28.145450 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:28.145542 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:28.145896 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 23:18:28.146044 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":63,"threads_started":1}
I20260924 23:18:28.147894 15504 heartbeater.cc:347] Connected to a master server at 127.1.4.254:38539
I20260924 23:18:28.147993 15504 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:28.148164 15504 heartbeater.cc:511] Master 127.1.4.254:38539 requested a full tablet report, sending...
I20260924 23:18:28.148991 15316 ts_manager.cc:194] Registered new tserver with Master: 9ebcbc07144b4cb3bf5d68848fa4e24d (127.1.4.193:38711)
I20260924 23:18:28.150219 15316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:59789
I20260924 23:18:28.151293 15504 heartbeater.cc:503] Master 127.1.4.254:38539 was elected leader, sending a full tablet report...
I20260924 23:18:28.158273 15512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.149932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.1.4.193:38711)
0924 23:18:28.154163 (+ 4231us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:28.154167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:28.154176 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:18:28.154210 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:28.154212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:28.154226 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:28.154311 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:28.154525 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.155648 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.157178 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.157867 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.157943 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.158001 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:28.158141 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 23:18:28.158180 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4196,"thread_start_us":69,"threads_started":1}
I20260924 23:18:28.158279 15511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.149679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:38711 (local address 127.0.0.1:54450)
0924 23:18:28.153894 (+ 4215us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:28.153944 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:28.154026 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:28.154351 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:28.154355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:28.154359 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:28.154444 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:28.154447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.155722 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.155723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:28.157086 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:28.157089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.157949 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.157950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:28.158040 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.158055 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 23:18:28.158122 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 23:18:28.158176 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4174,"thread_start_us":94,"threads_started":1}
I20260924 23:18:28.159075 15438 tablet_service.cc:1514] Processing DeleteTablet for tablet be1e2f4180cc43a793b5e25c00301555 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:18:27 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:54450
I20260924 23:18:28.159731 15513 tablet_replica.cc:330] stopping tablet replica
I20260924 23:18:28.159847 15513 raft_consensus.cc:2229] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:18:28.159967 15513 raft_consensus.cc:2258] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:18:28.160342 15513 ts_tablet_manager.cc:1905] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:18:28.162804 15513 ts_tablet_manager.cc:1918] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:18:28.162885 15513 log.cc:1196] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal/wals/be1e2f4180cc43a793b5e25c00301555
I20260924 23:18:28.163087 15513 ts_tablet_manager.cc:1939] T be1e2f4180cc43a793b5e25c00301555 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Deleting consensus metadata
I20260924 23:18:28.163667 15300 catalog_manager.cc:4841] TS 9ebcbc07144b4cb3bf5d68848fa4e24d (127.1.4.193:38711): tablet be1e2f4180cc43a793b5e25c00301555 (table test-table [id=915d785cc08144b281cb907295688e46]) successfully deleted
W20260924 23:18:28.227424 15509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:28.227626 15509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:28.227648 15509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:28.228816 15509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:28.228858 15509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:28.228883 15509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:18:28.230357 15509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:33269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=45435
--tserver_master_addrs=127.1.4.254:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:28.230595 15509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:28.230808 15509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:28.235749 15521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:28.235726 15519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:28.235715 15518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:28.235955 15509 server_base.cc:1034] running on GCE node
I20260924 23:18:28.236239 15509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:28.236475 15509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:28.237609 15509 hybrid_clock.cc:648] HybridClock initialized: now 1790291908237598 us; error 30 us; skew 500 ppm
Sep 24 23:18:28 dist-test-slave-s1hw krb5kdc[14740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291908, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:28.240186 15509 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:18:28.240617 15509 webserver.cc:460] Webserver started at http://127.1.4.194:45435/ using document root <none> and password file <none>
I20260924 23:18:28.240826 15509 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:28.240904 15509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:28.242161 15509 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:18:28.242666 15528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:28.242807 15509 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:28.242935 15509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "2c19ed3773504dba845328975d91e1f2"
format_stamp: "Formatted at 2026-09-24 23:18:27 on dist-test-slave-s1hw"
server_key: "ebc1cd33dd37c6f70d3f50f39076d21c"
server_key_iv: "f9b1fe9265d0ebf04c999d31828d59cc"
server_key_version: "encryptionkey@0"
I20260924 23:18:28.243259 15509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:28.252214 15509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:28.253871 15509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:28.253997 15509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:28.254195 15509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:28.254479 15509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:18:28.254510 15509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:28.254537 15509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:18:28.254561 15509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:28.259955 15509 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:33269
I20260924 23:18:28.260011 15641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:33269 every 8 connection(s)
I20260924 23:18:28.260360 15509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:18:28 dist-test-slave-s1hw krb5kdc[14740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291908, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:28.268646 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 15509
I20260924 23:18:28.268941 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:46597
--local_ip_for_outbound_sockets=127.1.4.195
--tserver_master_addrs=127.1.4.254:38539
--webserver_port=41039
--webserver_interface=127.1.4.195
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:18:28.269268 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.261020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:37393 (local address 127.1.4.254:38539)
0924 23:18:28.261079 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:28.261081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:28.261302 (+ 221us) server_negotiation.cc:407] Connection header received
0924 23:18:28.261769 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:28.261770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:28.261783 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:28.261831 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:28.262728 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.263256 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.264276 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.264488 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.264550 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.266587 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:28.266588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:28.266588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:28.266595 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:28.267860 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:28.268207 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:28.268208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:28.268209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:28.268263 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:28.268503 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:28.268505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:28.268505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:28.268667 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:28.268765 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:28.269049 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 23:18:28.269149 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 23:18:28.269359 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.261049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:37393)
0924 23:18:28.261188 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:28.261224 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:28.261693 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:28.261887 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:28.261890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:28.262465 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:28.262658 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:28.262663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.263321 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.263321 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:28.264184 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:28.264187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.264560 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.264561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:28.264658 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.265159 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:28.265167 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:28.266500 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:28.267922 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:28.267929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:28.267932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:28.268143 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:28.268312 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:28.268312 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:28.268313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:28.268399 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:28.268748 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:28.268753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:28.268912 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 23:18:28.269008 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 23:18:28.269178 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260924 23:18:28.269873 15642 heartbeater.cc:347] Connected to a master server at 127.1.4.254:38539
I20260924 23:18:28.269963 15642 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:28.270129 15642 heartbeater.cc:511] Master 127.1.4.254:38539 requested a full tablet report, sending...
I20260924 23:18:28.270516 15316 ts_manager.cc:194] Registered new tserver with Master: 2c19ed3773504dba845328975d91e1f2 (127.1.4.194:33269)
I20260924 23:18:28.270882 15316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:37393
W20260924 23:18:28.351357 15645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:28.351620 15645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:28.351651 15645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:28.352800 15645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:28.352846 15645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:28.352881 15645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:18:28.354351 15645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:46597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=41039
--tserver_master_addrs=127.1.4.254:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:28.354583 15645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:28.354787 15645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:28.359669 15650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:28.359669 15653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:28.359778 15645 server_base.cc:1034] running on GCE node
W20260924 23:18:28.359722 15651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:28.360091 15645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:28.360371 15645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:28.361536 15645 hybrid_clock.cc:648] HybridClock initialized: now 1790291908361514 us; error 35 us; skew 500 ppm
Sep 24 23:18:28 dist-test-slave-s1hw krb5kdc[14740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291908, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:28.363919 15645 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:18:28.364341 15645 webserver.cc:460] Webserver started at http://127.1.4.195:41039/ using document root <none> and password file <none>
I20260924 23:18:28.364534 15645 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:28.364596 15645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:28.365875 15645 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:28.366395 15660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:28.366539 15645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:28.366617 15645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "4045123d7a14416eb00f4978335ac53e"
format_stamp: "Formatted at 2026-09-24 23:18:27 on dist-test-slave-s1hw"
server_key: "15928bd7c6b248ae8fd97b6c11672e73"
server_key_iv: "82996ed268404362857123fa18e38d62"
server_key_version: "encryptionkey@0"
I20260924 23:18:28.366961 15645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:28.380115 15645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:28.382090 15645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:28.382228 15645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:28.382453 15645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:28.382745 15645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:18:28.382777 15645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:28.382807 15645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:18:28.382830 15645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:28.388372 15645 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:46597
I20260924 23:18:28.388433 15773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:46597 every 8 connection(s)
I20260924 23:18:28.388748 15645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:18:28.394084 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 15645
Sep 24 23:18:28 dist-test-slave-s1hw krb5kdc[14740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291908, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:28.397687 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.389443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:58643 (local address 127.1.4.254:38539)
0924 23:18:28.389493 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:28.389495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:28.389710 (+ 215us) server_negotiation.cc:407] Connection header received
0924 23:18:28.390243 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:28.390244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:28.390257 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:28.390308 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:28.391279 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.391795 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.392789 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.392978 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.393034 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.395166 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:28.395167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:28.395167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:28.395175 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:28.396454 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:28.396822 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:28.396823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:28.396823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:28.396877 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:28.397100 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:28.397101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:28.397101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:28.397204 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:28.397259 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:28.397519 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 23:18:28.397598 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:18:28.397900 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.389437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:58643)
0924 23:18:28.389592 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:28.389629 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:28.390141 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:28.390383 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:28.390387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:28.390982 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:28.391178 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:28.391183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.391877 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.391878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:28.392702 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:28.392705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.393048 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.393049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:28.393173 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.393650 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:28.393659 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:28.395079 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:28.396527 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:28.396534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:28.396536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:28.396758 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:28.396940 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:28.396940 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:28.396941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:28.397044 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:28.397256 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:28.397263 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:28.397400 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 23:18:28.397491 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 23:18:28.397647 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260924 23:18:28.398340 15774 heartbeater.cc:347] Connected to a master server at 127.1.4.254:38539
I20260924 23:18:28.398447 15774 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:28.398617 15774 heartbeater.cc:511] Master 127.1.4.254:38539 requested a full tablet report, sending...
I20260924 23:18:28.399116 15316 ts_manager.cc:194] Registered new tserver with Master: 4045123d7a14416eb00f4978335ac53e (127.1.4.195:46597)
I20260924 23:18:28.399659 15316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:58643
I20260924 23:18:28.400594 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.394480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.1.4.254:38539)
0924 23:18:28.394588 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:28.394590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:28.394598 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:18:28.394659 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:28.394660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:28.394671 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:28.394726 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:28.395227 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.395771 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.396473 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.396675 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.396727 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.397875 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:28.397876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:28.397876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:28.397883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:28.399535 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:28.399870 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:28.399871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:28.399871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:28.399931 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:28.400123 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:28.400124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:28.400125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:28.400244 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:28.400402 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:28.400451 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 23:18:28.400530 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":62,"threads_started":1}
I20260924 23:18:28.401254 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:18:28.405272 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.401582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.1.4.254:38539)
0924 23:18:28.401634 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:28.401635 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:28.402100 (+ 465us) server_negotiation.cc:407] Connection header received
0924 23:18:28.402159 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:28.402160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:28.402201 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:28.402269 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:18:28.402402 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.403318 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.404289 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.404667 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.404743 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.404873 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:18:28.405040 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:18:28.405127 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:28.405143 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 23:18:28.405163 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:18:28.407670 15316 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:32894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:18:28.409198 15316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:18:28.411306 15438 tablet_service.cc:1467] Processing CreateTablet for tablet ab6784d04a9c4fd6aaa1b3ef1a0bd502 (DEFAULT_TABLE table=test-table [id=ef04c890408b4165a29761e35a899862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:18:28.411522 15438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab6784d04a9c4fd6aaa1b3ef1a0bd502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:28.413089 15396 tablet_bootstrap.cc:492] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Bootstrap starting.
I20260924 23:18:28.413434 15396 tablet_bootstrap.cc:654] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:28.413988 15396 tablet_bootstrap.cc:492] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d: No bootstrap required, opened a new log
I20260924 23:18:28.414039 15396 ts_tablet_manager.cc:1397] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:28.414140 15396 raft_consensus.cc:348] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:28.414214 15396 raft_consensus.cc:374] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:28.414268 15396 raft_consensus.cc:729] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebcbc07144b4cb3bf5d68848fa4e24d, State: Initialized, Role: FOLLOWER
I20260924 23:18:28.414311 15396 consensus_queue.cc:260] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:28.414398 15396 ts_tablet_manager.cc:1428] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 23:18:28.415302 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.411478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47098 (local address 127.1.4.195:46597)
0924 23:18:28.411665 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:28.411671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:28.411681 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:18:28.411736 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:28.411739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:28.411755 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:28.411832 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:28.411969 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.412900 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.414301 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.414999 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.415061 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.415122 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:28.415186 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 23:18:28.415223 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":98,"threads_started":1}
I20260924 23:18:28.415315 15511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.410969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:46597 (local address 127.0.0.1:47098)
0924 23:18:28.411018 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:28.411028 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:28.411064 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:28.411850 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:28.411851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:28.411853 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:28.411897 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:28.411898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.412986 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.412988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:28.414030 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:28.414033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.415058 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.415059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:28.415116 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.415119 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:18:28.415171 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 23:18:28.415222 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":2}
I20260924 23:18:28.415331 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.411072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.1.4.194:33269)
0924 23:18:28.411724 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:28.411729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:28.411740 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:18:28.411779 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:28.411781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:28.411797 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:28.412076 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:28.412146 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.413207 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.414311 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.415007 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.415071 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.415125 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:28.415215 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 23:18:28.415250 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":91,"threads_started":1}
I20260924 23:18:28.415663 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.410971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:33269 (local address 127.0.0.1:48742)
0924 23:18:28.411169 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:28.411178 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:28.411215 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:28.411963 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:28.411964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:28.411966 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:28.412012 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:28.412013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.413317 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.413318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:28.414221 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:28.414225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.415080 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.415081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:28.415145 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.415148 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:18:28.415202 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 23:18:28.415248 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":16,"thread_start_us":103,"threads_started":1}
I20260924 23:18:28.416003 15708 tablet_service.cc:1467] Processing CreateTablet for tablet ab6784d04a9c4fd6aaa1b3ef1a0bd502 (DEFAULT_TABLE table=test-table [id=ef04c890408b4165a29761e35a899862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:18:28.416350 15708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab6784d04a9c4fd6aaa1b3ef1a0bd502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:28.416355 15576 tablet_service.cc:1467] Processing CreateTablet for tablet ab6784d04a9c4fd6aaa1b3ef1a0bd502 (DEFAULT_TABLE table=test-table [id=ef04c890408b4165a29761e35a899862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:18:28.416666 15576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab6784d04a9c4fd6aaa1b3ef1a0bd502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:28.419090 15784 tablet_bootstrap.cc:492] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e: Bootstrap starting.
I20260924 23:18:28.419122 15785 tablet_bootstrap.cc:492] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2: Bootstrap starting.
I20260924 23:18:28.419723 15785 tablet_bootstrap.cc:654] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:28.419735 15784 tablet_bootstrap.cc:654] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:28.420199 15784 log.cc:824] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:28.420199 15785 log.cc:824] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:28.420840 15785 tablet_bootstrap.cc:492] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2: No bootstrap required, opened a new log
I20260924 23:18:28.420902 15785 ts_tablet_manager.cc:1397] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 23:18:28.421052 15784 tablet_bootstrap.cc:492] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e: No bootstrap required, opened a new log
I20260924 23:18:28.421115 15784 ts_tablet_manager.cc:1397] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:18:28.422421 15785 raft_consensus.cc:348] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:28.422564 15785 raft_consensus.cc:374] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:28.422583 15785 raft_consensus.cc:729] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c19ed3773504dba845328975d91e1f2, State: Initialized, Role: FOLLOWER
I20260924 23:18:28.422686 15785 consensus_queue.cc:260] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:28.422672 15784 raft_consensus.cc:348] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:28.422839 15784 raft_consensus.cc:374] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:28.422878 15784 raft_consensus.cc:729] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4045123d7a14416eb00f4978335ac53e, State: Initialized, Role: FOLLOWER
I20260924 23:18:28.422969 15642 heartbeater.cc:503] Master 127.1.4.254:38539 was elected leader, sending a full tablet report...
I20260924 23:18:28.422993 15785 ts_tablet_manager.cc:1428] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:18:28.423259 15784 consensus_queue.cc:260] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:28.423573 15784 ts_tablet_manager.cc:1428] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:18:28.423583 15774 heartbeater.cc:503] Master 127.1.4.254:38539 was elected leader, sending a full tablet report...
I20260924 23:18:28.434990 15788 raft_consensus.cc:482] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:18:28.435105 15788 raft_consensus.cc:504] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:28.435501 15788 leader_election.cc:288] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4045123d7a14416eb00f4978335ac53e (127.1.4.195:46597), 9ebcbc07144b4cb3bf5d68848fa4e24d (127.1.4.193:38711)
I20260924 23:18:28.438344 15512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.435760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:51861 (local address 127.1.4.193:38711)
0924 23:18:28.435835 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:28.435837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:28.435848 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:18:28.435916 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:28.435917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:28.435928 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:28.435976 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:28.436103 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.436644 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.437639 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.438026 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.438090 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.438101 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:28.438242 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 23:18:28.438281 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 23:18:28.438344 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.435654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:38711 (local address 127.1.4.194:51861)
0924 23:18:28.435704 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:28.435717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:28.435857 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:28.435983 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:28.435984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:28.435985 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:28.436049 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:28.436051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.436715 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.436716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:28.437554 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:28.437557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.438089 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.438090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:28.438175 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.438178 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:18:28.438222 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:18:28.438266 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":33}
I20260924 23:18:28.438422 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.435768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:60619 (local address 127.1.4.195:46597)
0924 23:18:28.435817 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:28.435819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:28.435950 (+ 131us) server_negotiation.cc:407] Connection header received
0924 23:18:28.435989 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:28.435990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:28.436002 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:28.436046 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:28.436172 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.436638 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.437662 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.438111 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.438185 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.438195 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:28.438316 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 23:18:28.438354 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:18:28.438532 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.435661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:46597 (local address 127.1.4.194:60619)
0924 23:18:28.435876 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:28.435885 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:28.435944 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:28.436047 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:28.436047 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:28.436049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:28.436117 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:28.436119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.436716 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.436716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:28.437583 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:28.437586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.438189 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.438190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:28.438258 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.438262 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:18:28.438306 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:18:28.438345 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":92,"thread_start_us":104,"threads_started":1}
I20260924 23:18:28.438894 15728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6784d04a9c4fd6aaa1b3ef1a0bd502" candidate_uuid: "2c19ed3773504dba845328975d91e1f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4045123d7a14416eb00f4978335ac53e" is_pre_election: true
I20260924 23:18:28.438897 15458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6784d04a9c4fd6aaa1b3ef1a0bd502" candidate_uuid: "2c19ed3773504dba845328975d91e1f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" is_pre_election: true
I20260924 23:18:28.439050 15728 raft_consensus.cc:2454] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c19ed3773504dba845328975d91e1f2 in term 0.
I20260924 23:18:28.439050 15458 raft_consensus.cc:2454] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c19ed3773504dba845328975d91e1f2 in term 0.
I20260924 23:18:28.439263 15529 leader_election.cc:302] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c19ed3773504dba845328975d91e1f2, 9ebcbc07144b4cb3bf5d68848fa4e24d; no voters:
I20260924 23:18:28.439421 15788 raft_consensus.cc:2781] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:18:28.439479 15788 raft_consensus.cc:482] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:18:28.439496 15788 raft_consensus.cc:3037] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:28.440207 15788 raft_consensus.cc:504] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:28.440337 15788 leader_election.cc:288] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [CANDIDATE]: Term 1 election: Requested vote from peers 4045123d7a14416eb00f4978335ac53e (127.1.4.195:46597), 9ebcbc07144b4cb3bf5d68848fa4e24d (127.1.4.193:38711)
I20260924 23:18:28.440498 15728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6784d04a9c4fd6aaa1b3ef1a0bd502" candidate_uuid: "2c19ed3773504dba845328975d91e1f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4045123d7a14416eb00f4978335ac53e"
I20260924 23:18:28.440496 15458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6784d04a9c4fd6aaa1b3ef1a0bd502" candidate_uuid: "2c19ed3773504dba845328975d91e1f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d"
I20260924 23:18:28.440579 15458 raft_consensus.cc:3037] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:28.440578 15728 raft_consensus.cc:3037] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:28.441437 15728 raft_consensus.cc:2454] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c19ed3773504dba845328975d91e1f2 in term 1.
I20260924 23:18:28.441437 15458 raft_consensus.cc:2454] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c19ed3773504dba845328975d91e1f2 in term 1.
I20260924 23:18:28.441597 15529 leader_election.cc:302] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c19ed3773504dba845328975d91e1f2, 9ebcbc07144b4cb3bf5d68848fa4e24d; no voters:
I20260924 23:18:28.441706 15788 raft_consensus.cc:2781] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:18:28.441969 15788 raft_consensus.cc:686] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [term 1 LEADER]: Becoming Leader. State: Replica: 2c19ed3773504dba845328975d91e1f2, State: Running, Role: LEADER
I20260924 23:18:28.442070 15788 consensus_queue.cc:237] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:28.442900 15315 catalog_manager.cc:5495] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c19ed3773504dba845328975d91e1f2 (127.1.4.194). New cstate: current_term: 1 leader_uuid: "2c19ed3773504dba845328975d91e1f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260924 23:18:28.453977 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.451698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48744 (local address 127.1.4.194:33269)
0924 23:18:28.451811 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:28.451813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:28.451820 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:18:28.451850 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:28.451850 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:28.451867 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:28.451939 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:18:28.452020 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.452481 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.453231 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:28.453428 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:28.453489 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:28.453598 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:18:28.453693 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:18:28.453749 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:28.453790 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 23:18:28.453807 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 23:18:28.455955 15458 raft_consensus.cc:1261] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Refusing update from remote peer 2c19ed3773504dba845328975d91e1f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:18:28.455958 15728 raft_consensus.cc:1261] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e [term 1 FOLLOWER]: Refusing update from remote peer 2c19ed3773504dba845328975d91e1f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:18:28.456384 15788 consensus_queue.cc:1035] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:18:28.456535 15791 consensus_queue.cc:1035] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:18:28.458724 15794 mvcc.cc:205] Tried to move back new op lower bound from 7333035657032617984 to 7333035656979353600. Current Snapshot: MvccSnapshot[applied={T|T < 7333035657032617984}]
I20260924 23:18:28.459987 15796 mvcc.cc:205] Tried to move back new op lower bound from 7333035657032617984 to 7333035656979353600. Current Snapshot: MvccSnapshot[applied={T|T < 7333035657032617984 or (T in {7333035657032617984})}]
I20260924 23:18:28.460103 15795 mvcc.cc:205] Tried to move back new op lower bound from 7333035657032617984 to 7333035656979353600. Current Snapshot: MvccSnapshot[applied={T|T < 7333035657032617984 or (T in {7333035657032617984})}]
I20260924 23:18:28.461086 15316 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:32894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:18:28.461174 15316 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:32894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:18:28.461861 15316 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3: Sending DeleteTablet for 3 replicas of tablet ab6784d04a9c4fd6aaa1b3ef1a0bd502
I20260924 23:18:28.462175 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 15354
I20260924 23:18:28.462157 15576 tablet_service.cc:1514] Processing DeleteTablet for tablet ab6784d04a9c4fd6aaa1b3ef1a0bd502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:18:28 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:48742
I20260924 23:18:28.462280 15708 tablet_service.cc:1514] Processing DeleteTablet for tablet ab6784d04a9c4fd6aaa1b3ef1a0bd502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:18:28 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:47098
I20260924 23:18:28.462481 15804 tablet_replica.cc:330] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2: stopping tablet replica
I20260924 23:18:28.462666 15804 raft_consensus.cc:2229] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:18:28.462811 15804 raft_consensus.cc:2258] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:18:28.463919 15804 ts_tablet_manager.cc:1905] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:18:28.464267 15805 tablet_replica.cc:330] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e: stopping tablet replica
I20260924 23:18:28.464375 15805 raft_consensus.cc:2229] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:18:28.464481 15805 raft_consensus.cc:2258] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:18:28.466147 15804 ts_tablet_manager.cc:1918] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:18:28.466264 15804 log.cc:1196] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/ab6784d04a9c4fd6aaa1b3ef1a0bd502
I20260924 23:18:28.466569 15804 ts_tablet_manager.cc:1939] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 2c19ed3773504dba845328975d91e1f2: Deleting consensus metadata
I20260924 23:18:28.467008 15302 catalog_manager.cc:4841] TS 2c19ed3773504dba845328975d91e1f2 (127.1.4.194:33269): tablet ab6784d04a9c4fd6aaa1b3ef1a0bd502 (table test-table [id=ef04c890408b4165a29761e35a899862]) successfully deleted
W20260924 23:18:28.467211 15300 connection.cc:535] client connection to 127.1.4.193:38711 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 23:18:28.467303 15300 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 23:18:28.467578 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 15509
I20260924 23:18:28.470085 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.469926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:38711 (local address 127.0.0.1:54462)
0924 23:18:28.470004 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:28.470053 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:38711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 23:18:28.470209 15805 ts_tablet_manager.cc:1905] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 23:18:28.470230 15300 catalog_manager.cc:4584] TS 9ebcbc07144b4cb3bf5d68848fa4e24d (127.1.4.193:38711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab6784d04a9c4fd6aaa1b3ef1a0bd502: Network error: Client connection negotiation failed: client connection to 127.1.4.193:38711: connect: Connection refused (error 111)
I20260924 23:18:28.471952 15805 ts_tablet_manager.cc:1918] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:18:28.472020 15805 log.cc:1196] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/ab6784d04a9c4fd6aaa1b3ef1a0bd502
I20260924 23:18:28.472260 15805 ts_tablet_manager.cc:1939] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 4045123d7a14416eb00f4978335ac53e: Deleting consensus metadata
I20260924 23:18:28.472647 15301 catalog_manager.cc:4841] TS 4045123d7a14416eb00f4978335ac53e (127.1.4.195:46597): tablet ab6784d04a9c4fd6aaa1b3ef1a0bd502 (table test-table [id=ef04c890408b4165a29761e35a899862]) successfully deleted
I20260924 23:18:28.474589 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 15645
I20260924 23:18:28.479149 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 15283
I20260924 23:18:28.484071 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38539
--webserver_interface=127.1.4.254
--webserver_port=39367
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:38539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:28.564770 15807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:28.564992 15807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:28.565016 15807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:28.566118 15807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:18:28.566152 15807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:28.566164 15807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:28.566175 15807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:18:28.566185 15807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:18:28.567617 15807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:38539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=39367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:28.567814 15807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:28.567992 15807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:18:28.573069 15807 server_base.cc:1034] running on GCE node
W20260924 23:18:28.573019 15813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:28.573060 15812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:28.573019 15815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:28.573436 15807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:28.573716 15807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:28.574859 15807 hybrid_clock.cc:648] HybridClock initialized: now 1790291908574846 us; error 24 us; skew 500 ppm
Sep 24 23:18:28 dist-test-slave-s1hw krb5kdc[14740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291908, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:28.577386 15807 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:18:28.577807 15807 webserver.cc:460] Webserver started at http://127.1.4.254:39367/ using document root <none> and password file <none>
I20260924 23:18:28.577991 15807 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:28.578032 15807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:28.579342 15807 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:18:28.580076 15822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:28.580241 15807 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:28.580324 15807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "d5a84caaa04d447f9f88cf67715793a3"
format_stamp: "Formatted at 2026-09-24 23:18:26 on dist-test-slave-s1hw"
server_key: "a82b652222d34e57bb56fe4e48e876eb"
server_key_iv: "c56d17398e90b559ece43978b4c386bd"
server_key_version: "encryptionkey@0"
I20260924 23:18:28.580665 15807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:28.602388 15807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:28.604458 15807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:28.604595 15807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:28.608413 15807 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:38539
I20260924 23:18:28.608445 15874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:38539 every 8 connection(s)
I20260924 23:18:28.608820 15807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:18:28.610380 15875 sys_catalog.cc:263] Verifying existing consensus state
I20260924 23:18:28.610983 15875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3: Bootstrap starting.
I20260924 23:18:28.618884 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 15807
I20260924 23:18:28.618878 15875 log.cc:824] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:28.619194 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:38711
--local_ip_for_outbound_sockets=127.1.4.193
--tserver_master_addrs=127.1.4.254:38539
--webserver_port=41847
--webserver_interface=127.1.4.193
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:18:28.621335 15875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:18:28.621593 15875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3: Bootstrap complete.
I20260924 23:18:28.623732 15875 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } }
I20260924 23:18:28.623963 15875 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5a84caaa04d447f9f88cf67715793a3, State: Initialized, Role: FOLLOWER
I20260924 23:18:28.624137 15875 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } }
I20260924 23:18:28.624222 15875 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:18:28.624271 15875 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:18:28.624320 15875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 2 FOLLOWER]: Advancing to term 3
I20260924 23:18:28.625278 15875 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } }
I20260924 23:18:28.625372 15875 leader_election.cc:302] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5a84caaa04d447f9f88cf67715793a3; no voters:
I20260924 23:18:28.625525 15875 leader_election.cc:288] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 23:18:28.625589 15879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 23:18:28.625738 15879 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [term 3 LEADER]: Becoming Leader. State: Replica: d5a84caaa04d447f9f88cf67715793a3, State: Running, Role: LEADER
I20260924 23:18:28.625773 15875 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:18:28.625852 15879 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } }
I20260924 23:18:28.626211 15879 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5a84caaa04d447f9f88cf67715793a3. Latest consensus state: current_term: 3 leader_uuid: "d5a84caaa04d447f9f88cf67715793a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } } }
I20260924 23:18:28.626282 15879 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [sys.catalog]: This master's current role is: LEADER
I20260924 23:18:28.626377 15880 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d5a84caaa04d447f9f88cf67715793a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a84caaa04d447f9f88cf67715793a3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38539 } } }
I20260924 23:18:28.626456 15880 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a84caaa04d447f9f88cf67715793a3 [sys.catalog]: This master's current role is: LEADER
I20260924 23:18:28.626549 15885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:18:28.627445 15885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:18:28.627646 15885 catalog_manager.cc:1259] Loaded cluster ID: 975588b2e67e42efab40d3531d855e07
I20260924 23:18:28.627875 15885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 23:18:28.628144 15885 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f18f3f609ad google::LogMessage::Fail() at ??:0
@ 0x7f18f3f6388a google::LogMessage::SendToLog() at ??:0
@ 0x7f18f3f604de google::LogMessage::Flush() at ??:0
@ 0x7f18f3f61a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f18fbe810f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f18fbe8a8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f18f5e2e0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f18f5e23954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f18f6c876db start_thread at ??:0
@ 0x7f18f2a5e61f clone at ??:0
W20260924 23:18:28.709020 15878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:28.709225 15878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:28.709262 15878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:28.710404 15878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:28.710469 15878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:28.710506 15878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:18:28.711977 15878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:38711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=41847
--tserver_master_addrs=127.1.4.254:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:28.712218 15878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:28.712428 15878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:28.717172 15911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:28.717342 15912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:28.717464 15914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:28.717592 15878 server_base.cc:1034] running on GCE node
I20260924 23:18:28.717798 15878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:28.718075 15878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:28.719256 15878 hybrid_clock.cc:648] HybridClock initialized: now 1790291908719236 us; error 31 us; skew 500 ppm
Sep 24 23:18:28 dist-test-slave-s1hw krb5kdc[14740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291908, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:28.722190 15878 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:18:28.722744 15878 webserver.cc:460] Webserver started at http://127.1.4.193:41847/ using document root <none> and password file <none>
I20260924 23:18:28.723014 15878 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:28.723084 15878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:28.724782 15878 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:28.725497 15921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:28.725652 15878 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:18:28.725745 15878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d"
format_stamp: "Formatted at 2026-09-24 23:18:27 on dist-test-slave-s1hw"
server_key: "5eea3dae2bc7398cf7147c93e64fe921"
server_key_iv: "413335f1a701b9f55440039e2d76a63a"
server_key_version: "encryptionkey@0"
I20260924 23:18:28.726178 15878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:28.769485 15878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:28.772104 15878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:28.772284 15878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:28.772567 15878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:28.773140 15928 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 23:18:28.774652 15878 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 23:18:28.774715 15878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 23:18:28.774750 15878 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 23:18:28.775810 15878 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 23:18:28.775863 15878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 23:18:28.775910 15928 tablet_bootstrap.cc:492] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Bootstrap starting.
I20260924 23:18:28.782964 15878 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:38711
I20260924 23:18:28.783140 16035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:38711 every 8 connection(s)
I20260924 23:18:28.783471 15878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:18:28.784726 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 15878
I20260924 23:18:28.785017 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:33269
--local_ip_for_outbound_sockets=127.1.4.194
--tserver_master_addrs=127.1.4.254:38539
--webserver_port=45435
--webserver_interface=127.1.4.194
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:18:28.786060 15928 log.cc:824] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:28.788709 15928 tablet_bootstrap.cc:492] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:18:28.789004 15928 tablet_bootstrap.cc:492] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Bootstrap complete.
I20260924 23:18:28.790094 15928 ts_tablet_manager.cc:1397] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260924 23:18:28.791610 15928 raft_consensus.cc:348] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:28.791950 15928 raft_consensus.cc:729] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebcbc07144b4cb3bf5d68848fa4e24d, State: Initialized, Role: FOLLOWER
I20260924 23:18:28.792127 15928 consensus_queue.cc:260] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:28.792440 15928 ts_tablet_manager.cc:1428] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d: Time spent starting tablet: real 0.002s user 0.000s sys 0.005s
W20260924 23:18:28.860137 16038 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:18:28.784238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:56787)
0924 23:18:28.784477 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:28.784534 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:28.785162 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:28.859947 (+ 74785us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":111,"threads_started":1}
W20260924 23:18:28.860430 16036 heartbeater.cc:644] Failed to heartbeat to 127.1.4.254:38539 (0 consecutive failures): Network error: Failed to ping master at 127.1.4.254:38539: Client connection negotiation failed: client connection to 127.1.4.254:38539: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 23:18:28.860738 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.860632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:49389)
0924 23:18:28.860678 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:28.860716 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 23:18:28.861155 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.861008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:47085)
0924 23:18:28.861094 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:28.861127 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 23:18:28.861285 16036 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 23:18:28.871603 16039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:28.871841 16039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:28.871876 16039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:28.873167 16039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:28.873229 16039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:28.873268 16039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:18:28.874807 16039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:33269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=45435
--tserver_master_addrs=127.1.4.254:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:28.875078 16039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:28.875305 16039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:28.880731 16046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:28.880743 16047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:28.880746 16049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:28.880942 16039 server_base.cc:1034] running on GCE node
I20260924 23:18:28.881114 16039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:28.881341 16039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:28.882493 16039 hybrid_clock.cc:648] HybridClock initialized: now 1790291908882483 us; error 44 us; skew 500 ppm
Sep 24 23:18:28 dist-test-slave-s1hw krb5kdc[14740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291908, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:28.885043 16039 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:18:28.885479 16039 webserver.cc:460] Webserver started at http://127.1.4.194:45435/ using document root <none> and password file <none>
I20260924 23:18:28.885684 16039 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:28.885733 16039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:28.887115 16039 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:28.887751 16056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:28.887952 16039 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:28.888038 16039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "2c19ed3773504dba845328975d91e1f2"
format_stamp: "Formatted at 2026-09-24 23:18:27 on dist-test-slave-s1hw"
server_key: "ebc1cd33dd37c6f70d3f50f39076d21c"
server_key_iv: "f9b1fe9265d0ebf04c999d31828d59cc"
server_key_version: "encryptionkey@0"
I20260924 23:18:28.888412 16039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:28.901539 16039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:28.903460 16039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:28.903599 16039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:28.903832 16039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:28.904131 16039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:18:28.904165 16039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:28.904192 16039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:18:28.904210 16039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:28.909795 16039 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:33269
I20260924 23:18:28.909936 16169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:33269 every 8 connection(s)
I20260924 23:18:28.910193 16039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:18:28.910751 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 16039
I20260924 23:18:28.910990 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:46597
--local_ip_for_outbound_sockets=127.1.4.195
--tserver_master_addrs=127.1.4.254:38539
--webserver_port=41039
--webserver_interface=127.1.4.195
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:18:28.911383 16172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.910922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:47401)
0924 23:18:28.911085 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:28.911179 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
W20260924 23:18:28.911875 16170 heartbeater.cc:644] Failed to heartbeat to 127.1.4.254:38539 (0 consecutive failures): Network error: Failed to ping master at 127.1.4.254:38539: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
I20260924 23:18:28.912194 16172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.912086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:49261)
0924 23:18:28.912135 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:28.912169 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 23:18:28.912451 16172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:28.912375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:34869)
0924 23:18:28.912414 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:28.912435 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 23:18:28.912573 16170 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 23:18:28.992004 16173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:28.992231 16173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:28.992256 16173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:28.993381 16173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:28.993413 16173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:28.993438 16173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:18:28.994831 16173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:46597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=41039
--tserver_master_addrs=127.1.4.254:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:28.995011 16173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:28.995180 16173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:29.000101 16179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:29.000185 16173 server_base.cc:1034] running on GCE node
W20260924 23:18:29.000108 16178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:29.000108 16181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:29.000484 16173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:29.000705 16173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:29.001840 16173 hybrid_clock.cc:648] HybridClock initialized: now 1790291909001826 us; error 28 us; skew 500 ppm
Sep 24 23:18:29 dist-test-slave-s1hw krb5kdc[14740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291909, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:29.004208 16173 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:18:29.004623 16173 webserver.cc:460] Webserver started at http://127.1.4.195:41039/ using document root <none> and password file <none>
I20260924 23:18:29.004940 16173 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:29.005003 16173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:29.006348 16173 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:29.006947 16188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:29.007113 16173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:18:29.007177 16173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "4045123d7a14416eb00f4978335ac53e"
format_stamp: "Formatted at 2026-09-24 23:18:27 on dist-test-slave-s1hw"
server_key: "15928bd7c6b248ae8fd97b6c11672e73"
server_key_iv: "82996ed268404362857123fa18e38d62"
server_key_version: "encryptionkey@0"
I20260924 23:18:29.007568 16173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:29.019028 16173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:29.021181 16173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:29.021293 16173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:29.021486 16173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:29.021742 16173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:18:29.021775 16173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:29.021803 16173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:18:29.021819 16173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:29.027279 16173 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:46597
I20260924 23:18:29.027325 16301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:46597 every 8 connection(s)
I20260924 23:18:29.027671 16173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:18:29.028631 16304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:29.028281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:40165)
0924 23:18:29.028422 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:29.028491 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
W20260924 23:18:29.028983 16302 heartbeater.cc:644] Failed to heartbeat to 127.1.4.254:38539 (0 consecutive failures): Network error: Failed to ping master at 127.1.4.254:38539: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
I20260924 23:18:29.029209 16304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:29.029134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:32823)
0924 23:18:29.029170 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:29.029194 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 23:18:29.029415 16304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:29.029345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:49621)
0924 23:18:29.029373 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:29.029390 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 23:18:29.029493 16302 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 23:18:29.036021 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 16173
I20260924 23:18:29.861915 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:29.861639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:50751)
0924 23:18:29.861813 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:29.861850 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260924 23:18:29.913148 16307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:29.912925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:60333)
0924 23:18:29.913055 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:29.913096 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260924 23:18:30.030128 16308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:30.029856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:59159)
0924 23:18:30.029988 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:30.030060 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260924 23:18:30.118777 16309 raft_consensus.cc:482] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:18:30.118916 16309 raft_consensus.cc:504] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:30.119297 16309 leader_election.cc:288] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c19ed3773504dba845328975d91e1f2 (127.1.4.194:33269), 4045123d7a14416eb00f4978335ac53e (127.1.4.195:46597)
Sep 24 23:18:30 dist-test-slave-s1hw krb5kdc[14740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291908, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.195@KRBTEST.COM
Sep 24 23:18:30 dist-test-slave-s1hw krb5kdc[14740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291908, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.194@KRBTEST.COM
I20260924 23:18:30.130071 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:30.119483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:46597 (local address 127.1.4.193:49119)
0924 23:18:30.119534 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:30.119559 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:30.119613 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:30.120481 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:30.120483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:30.120883 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:30.121072 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:30.121077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:30.122125 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:30.122126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:30.122956 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:30.122959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:30.123582 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:30.123583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:30.123662 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:30.124201 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:30.124211 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:30.125786 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:30.128030 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:30.128033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:30.128035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:30.128400 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:30.128755 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:30.128756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:30.128756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:30.128946 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:30.129494 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:30.129502 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:30.129672 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 23:18:30.129783 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 23:18:30.129958 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":45}
I20260924 23:18:30.130122 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:30.119646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:49119 (local address 127.1.4.195:46597)
0924 23:18:30.119775 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:30.119786 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:30.119815 (+ 29us) server_negotiation.cc:407] Connection header received
0924 23:18:30.120263 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:30.120265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:30.120304 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:30.120425 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:30.121190 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:30.122035 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:30.123043 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:30.123477 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:30.123577 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:30.125972 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:30.125978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:30.125978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:30.125989 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:30.127955 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:30.128498 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:30.128502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:30.128502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:30.128636 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:30.129049 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:30.129050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:30.129050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:30.129415 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:30.129506 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:30.129860 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 23:18:30.130008 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":51,"threads_started":1}
I20260924 23:18:30.130122 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:30.119597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:51855 (local address 127.1.4.194:33269)
0924 23:18:30.119738 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:30.119751 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:30.119783 (+ 32us) server_negotiation.cc:407] Connection header received
0924 23:18:30.120253 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:30.120256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:30.120305 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:30.120435 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:30.121179 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:30.122026 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:30.123035 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:30.123462 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:30.123553 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:30.126160 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:30.126165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:30.126165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:30.126177 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:30.127918 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:30.128499 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:30.128502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:30.128502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:30.128634 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:30.129049 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:30.129050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:30.129051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:30.129415 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:30.129503 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:30.129885 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 23:18:30.130008 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":51,"threads_started":1}
I20260924 23:18:30.130137 16310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:30.119484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:33269 (local address 127.1.4.193:51855)
0924 23:18:30.119648 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:30.119656 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:30.119698 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:30.120524 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:30.120525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:30.120885 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:30.121072 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:30.121077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:30.122117 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:30.122118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:30.122955 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:30.122957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:30.123543 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:30.123544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:30.123675 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:30.124236 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:30.124240 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:30.126067 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:30.128024 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:30.128032 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:30.128035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:30.128430 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:30.128737 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:30.128738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:30.128739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:30.128946 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:30.129503 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:30.129503 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:30.129672 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 23:18:30.129783 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 23:18:30.130026 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":167,"thread_start_us":59,"threads_started":1}
W20260924 23:18:30.131482 15924 leader_election.cc:341] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c19ed3773504dba845328975d91e1f2 (127.1.4.194:33269): Not found: Tablet not found: ab6784d04a9c4fd6aaa1b3ef1a0bd502
W20260924 23:18:30.131592 15923 leader_election.cc:341] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4045123d7a14416eb00f4978335ac53e (127.1.4.195:46597): Not found: Tablet not found: ab6784d04a9c4fd6aaa1b3ef1a0bd502
I20260924 23:18:30.131626 15923 leader_election.cc:302] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ebcbc07144b4cb3bf5d68848fa4e24d; no voters: 2c19ed3773504dba845328975d91e1f2, 4045123d7a14416eb00f4978335ac53e
I20260924 23:18:30.131743 16309 raft_consensus.cc:2726] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:18:30.862710 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:30.862448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:38855)
0924 23:18:30.862612 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:30.862662 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":70,"threads_started":1}
I20260924 23:18:30.913887 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:30.913660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:41701)
0924 23:18:30.913802 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:30.913843 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260924 23:18:31.030900 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:31.030635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:37181)
0924 23:18:31.030777 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:31.030821 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":60,"threads_started":1}
I20260924 23:18:31.863566 16316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:31.863251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:54657)
0924 23:18:31.863468 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:31.863519 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":68,"threads_started":1}
I20260924 23:18:31.914633 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:31.914419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:46367)
0924 23:18:31.914542 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:31.914584 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260924 23:18:32.031709 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:32.031469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:42723)
0924 23:18:32.031608 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:32.031665 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":58,"threads_started":1}
I20260924 23:18:32.225005 16319 raft_consensus.cc:482] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:18:32.225100 16319 raft_consensus.cc:504] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:32.225250 16319 leader_election.cc:288] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c19ed3773504dba845328975d91e1f2 (127.1.4.194:33269), 4045123d7a14416eb00f4978335ac53e (127.1.4.195:46597)
W20260924 23:18:32.225629 15923 leader_election.cc:341] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4045123d7a14416eb00f4978335ac53e (127.1.4.195:46597): Not found: Tablet not found: ab6784d04a9c4fd6aaa1b3ef1a0bd502
W20260924 23:18:32.225708 15924 leader_election.cc:341] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c19ed3773504dba845328975d91e1f2 (127.1.4.194:33269): Not found: Tablet not found: ab6784d04a9c4fd6aaa1b3ef1a0bd502
I20260924 23:18:32.225795 15924 leader_election.cc:302] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ebcbc07144b4cb3bf5d68848fa4e24d; no voters: 2c19ed3773504dba845328975d91e1f2, 4045123d7a14416eb00f4978335ac53e
I20260924 23:18:32.225878 16319 raft_consensus.cc:2726] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:18:32.864341 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:32.864105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:59655)
0924 23:18:32.864247 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:32.864293 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260924 23:18:32.915340 16321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:32.915137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:38377)
0924 23:18:32.915252 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:32.915295 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260924 23:18:33.032361 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:33.032134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:44833)
0924 23:18:33.032277 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:33.032318 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260924 23:18:33.865209 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:33.864956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:35255)
0924 23:18:33.865111 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:33.865162 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260924 23:18:33.916007 16324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:33.915772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:49765)
0924 23:18:33.915920 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:33.915963 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260924 23:18:33.942098 16325 raft_consensus.cc:482] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:18:33.942193 16325 raft_consensus.cc:504] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:33.942329 16325 leader_election.cc:288] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c19ed3773504dba845328975d91e1f2 (127.1.4.194:33269), 4045123d7a14416eb00f4978335ac53e (127.1.4.195:46597)
W20260924 23:18:33.942713 15923 leader_election.cc:341] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4045123d7a14416eb00f4978335ac53e (127.1.4.195:46597): Not found: Tablet not found: ab6784d04a9c4fd6aaa1b3ef1a0bd502
W20260924 23:18:33.942806 15924 leader_election.cc:341] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c19ed3773504dba845328975d91e1f2 (127.1.4.194:33269): Not found: Tablet not found: ab6784d04a9c4fd6aaa1b3ef1a0bd502
I20260924 23:18:33.942878 15924 leader_election.cc:302] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ebcbc07144b4cb3bf5d68848fa4e24d; no voters: 2c19ed3773504dba845328975d91e1f2, 4045123d7a14416eb00f4978335ac53e
I20260924 23:18:33.942966 16325 raft_consensus.cc:2726] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:18:34.033013 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:34.032781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:57805)
0924 23:18:34.032926 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:34.032969 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260924 23:18:34.865906 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:34.865686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:58855)
0924 23:18:34.865817 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:34.865855 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260924 23:18:34.916720 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:34.916488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:42535)
0924 23:18:34.916624 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:34.916673 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260924 23:18:35.033730 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:35.033500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:49921)
0924 23:18:35.033628 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:35.033674 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260924 23:18:35.866567 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:35.866380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:34613)
0924 23:18:35.866490 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:35.866523 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260924 23:18:35.917429 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:35.917153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:39887)
0924 23:18:35.917296 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:35.917343 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260924 23:18:36.034392 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:36.034172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:41967)
0924 23:18:36.034301 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:36.034343 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260924 23:18:36.867262 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:36.867046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:52119)
0924 23:18:36.867171 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:36.867215 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260924 23:18:36.918133 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:36.917904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:36065)
0924 23:18:36.918042 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:36.918087 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260924 23:18:37.035084 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:37.034854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:45225)
0924 23:18:37.034992 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:37.035037 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":58,"threads_started":1}
I20260924 23:18:37.868060 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:37.867797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:42199)
0924 23:18:37.867966 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:37.868015 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":67,"threads_started":1}
I20260924 23:18:37.918809 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:37.918594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:55189)
0924 23:18:37.918720 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:37.918762 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
I20260924 23:18:38.035805 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:38.035572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:40521)
0924 23:18:38.035713 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:38.035758 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260924 23:18:38.050942 16339 raft_consensus.cc:482] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:18:38.051034 16339 raft_consensus.cc:504] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:38.051198 16339 leader_election.cc:288] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c19ed3773504dba845328975d91e1f2 (127.1.4.194:33269), 4045123d7a14416eb00f4978335ac53e (127.1.4.195:46597)
W20260924 23:18:38.051535 15923 leader_election.cc:341] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4045123d7a14416eb00f4978335ac53e (127.1.4.195:46597): Not found: Tablet not found: ab6784d04a9c4fd6aaa1b3ef1a0bd502
W20260924 23:18:38.051651 15924 leader_election.cc:341] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c19ed3773504dba845328975d91e1f2 (127.1.4.194:33269): Not found: Tablet not found: ab6784d04a9c4fd6aaa1b3ef1a0bd502
I20260924 23:18:38.051721 15924 leader_election.cc:302] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ebcbc07144b4cb3bf5d68848fa4e24d; no voters: 2c19ed3773504dba845328975d91e1f2, 4045123d7a14416eb00f4978335ac53e
I20260924 23:18:38.051795 16339 raft_consensus.cc:2726] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:18:38.868950 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:38.868655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:53551)
0924 23:18:38.868815 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:38.868905 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":65,"threads_started":1}
I20260924 23:18:38.919644 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:38.919402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:35937)
0924 23:18:38.919552 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:38.919597 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
I20260924 23:18:39.036576 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:39.036325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:33811)
0924 23:18:39.036477 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:39.036529 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260924 23:18:39.869721 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:39.869483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:60367)
0924 23:18:39.869615 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:39.869664 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260924 23:18:39.920287 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:39.920056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:60165)
0924 23:18:39.920203 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:39.920240 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260924 23:18:40.037302 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:40.037054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:36045)
0924 23:18:40.037205 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:40.037251 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260924 23:18:40.870507 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:40.870229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:56327)
0924 23:18:40.870409 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:40.870459 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":86,"threads_started":1}
I20260924 23:18:40.921058 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:40.920849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:54297)
0924 23:18:40.920968 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:40.921011 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260924 23:18:41.038113 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:41.037842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:46605)
0924 23:18:41.038010 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:41.038061 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":79,"threads_started":1}
I20260924 23:18:41.672781 16349 raft_consensus.cc:482] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:18:41.672889 16349 raft_consensus.cc:504] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c19ed3773504dba845328975d91e1f2" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 33269 } } peers { permanent_uuid: "4045123d7a14416eb00f4978335ac53e" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 46597 } } peers { permanent_uuid: "9ebcbc07144b4cb3bf5d68848fa4e24d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 38711 } }
I20260924 23:18:41.673103 16349 leader_election.cc:288] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c19ed3773504dba845328975d91e1f2 (127.1.4.194:33269), 4045123d7a14416eb00f4978335ac53e (127.1.4.195:46597)
W20260924 23:18:41.673432 15923 leader_election.cc:341] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4045123d7a14416eb00f4978335ac53e (127.1.4.195:46597): Not found: Tablet not found: ab6784d04a9c4fd6aaa1b3ef1a0bd502
W20260924 23:18:41.673498 15924 leader_election.cc:341] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c19ed3773504dba845328975d91e1f2 (127.1.4.194:33269): Not found: Tablet not found: ab6784d04a9c4fd6aaa1b3ef1a0bd502
I20260924 23:18:41.673542 15924 leader_election.cc:302] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ebcbc07144b4cb3bf5d68848fa4e24d; no voters: 2c19ed3773504dba845328975d91e1f2, 4045123d7a14416eb00f4978335ac53e
I20260924 23:18:41.673619 16349 raft_consensus.cc:2726] T ab6784d04a9c4fd6aaa1b3ef1a0bd502 P 9ebcbc07144b4cb3bf5d68848fa4e24d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:18:41.871270 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:41.871028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:59697)
0924 23:18:41.871172 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:41.871212 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260924 23:18:41.921728 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:41.921505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:33903)
0924 23:18:41.921634 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:41.921680 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260924 23:18:42.038874 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:42.038615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:52575)
0924 23:18:42.038777 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:42.038824 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20260924 23:18:42.872067 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:42.871817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:57155)
0924 23:18:42.871967 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:42.872015 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260924 23:18:42.922379 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:42.922167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:33841)
0924 23:18:42.922289 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:42.922331 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260924 23:18:43.039646 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:43.039422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:46359)
0924 23:18:43.039551 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:43.039597 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260924 23:18:43.872879 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:43.872629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.193:48881)
0924 23:18:43.872784 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:43.872831 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
I20260924 23:18:43.923058 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:43.922852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.194:50081)
0924 23:18:43.922973 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:43.923012 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260924 23:18:44.040470 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:44.040233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:38539 (local address 127.1.4.195:55719)
0924 23:18:44.040371 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:44.040420 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260924 23:18:44.045434 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 15878
I20260924 23:18:44.049889 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 16039
I20260924 23:18:44.053464 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 16173
2026-09-24T23:18:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18235 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:18:44 dist-test-slave-s1hw krb5kdc[16361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:18:44 dist-test-slave-s1hw krb5kdc[16361](info): set up 2 sockets
Sep 24 23:18:44 dist-test-slave-s1hw krb5kdc[16361](info): commencing operation
krb5kdc: starting...
W20260924 23:18:45.007347 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:18:45 dist-test-slave-s1hw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:18:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:18:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:18:45.118003 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:33901
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:33901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:45.199644 16374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:45.199904 16374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:45.199937 16374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:45.201115 16374 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 23:18:45.201157 16374 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 23:18:45.201174 16374 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 23:18:45.201188 16374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:18:45.201205 16374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:45.201222 16374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:45.201238 16374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:18:45.201260 16374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:18:45.202718 16374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:33901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:33901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:45.202972 16374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:45.203178 16374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:45.208221 16380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:45.208204 16382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:45.208314 16374 server_base.cc:1034] running on GCE node
W20260924 23:18:45.208206 16379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:45.208617 16374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:45.208871 16374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:45.210002 16374 hybrid_clock.cc:648] HybridClock initialized: now 1790291925209988 us; error 27 us; skew 500 ppm
Sep 24 23:18:45 dist-test-slave-s1hw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291925, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:45.215632 16374 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:18:45.216089 16374 webserver.cc:460] Webserver started at http://127.1.4.254:42041/ using document root <none> and password file <none>
I20260924 23:18:45.216293 16374 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:45.216337 16374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:45.216456 16374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:18:45.217526 16374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "67ac7572fd034c6eae1337ee7ae185cf"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "61c410df8ed17178767ab6a3352d484f"
server_key_iv: "59dcb1fd36487ba29d48dc703e029ac4"
server_key_version: "encryptionkey@0"
I20260924 23:18:45.217850 16374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "67ac7572fd034c6eae1337ee7ae185cf"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "61c410df8ed17178767ab6a3352d484f"
server_key_iv: "59dcb1fd36487ba29d48dc703e029ac4"
server_key_version: "encryptionkey@0"
I20260924 23:18:45.219067 16374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:18:45.219830 16389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:45.219969 16374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:45.220054 16374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "67ac7572fd034c6eae1337ee7ae185cf"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "61c410df8ed17178767ab6a3352d484f"
server_key_iv: "59dcb1fd36487ba29d48dc703e029ac4"
server_key_version: "encryptionkey@0"
I20260924 23:18:45.220115 16374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:45.235800 16374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:45.237514 16374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:45.237653 16374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:45.241433 16374 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:33901
I20260924 23:18:45.241478 16441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:33901 every 8 connection(s)
I20260924 23:18:45.241792 16374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:18:45.242556 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 16374
I20260924 23:18:45.242652 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:18:45.242686 16442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:45.242870 1043 external_mini_cluster.cc:1428] Setting key 4bee3af5a4fb5b525c509c891f076265
I20260924 23:18:45.245710 16442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Bootstrap starting.
I20260924 23:18:45.246408 16442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:45.246690 16442 log.cc:824] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:45.247330 16442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: No bootstrap required, opened a new log
Sep 24 23:18:45 dist-test-slave-s1hw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:45.249375 16442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } }
I20260924 23:18:45.249543 16442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:45.249585 16442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67ac7572fd034c6eae1337ee7ae185cf, State: Initialized, Role: FOLLOWER
I20260924 23:18:45.249713 16442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } }
I20260924 23:18:45.249796 16442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:18:45.249835 16442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:18:45.249883 16442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:45.250463 16442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } }
I20260924 23:18:45.250566 16442 leader_election.cc:302] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67ac7572fd034c6eae1337ee7ae185cf; no voters:
I20260924 23:18:45.250705 16442 leader_election.cc:288] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:18:45.250734 16447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:18:45.250828 16447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 1 LEADER]: Becoming Leader. State: Replica: 67ac7572fd034c6eae1337ee7ae185cf, State: Running, Role: LEADER
I20260924 23:18:45.250970 16442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:18:45.250969 16447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } }
I20260924 23:18:45.252503 16448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67ac7572fd034c6eae1337ee7ae185cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } } }
I20260924 23:18:45.252610 16448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [sys.catalog]: This master's current role is: LEADER
W20260924 23:18:45.252712 16462 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:18:45.252772 16462 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:18:45.252795 16449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67ac7572fd034c6eae1337ee7ae185cf. Latest consensus state: current_term: 1 leader_uuid: "67ac7572fd034c6eae1337ee7ae185cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } } }
I20260924 23:18:45.252871 16464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:18:45.253039 16464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:18:45.252851 16449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [sys.catalog]: This master's current role is: LEADER
I20260924 23:18:45.254400 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:45.243385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.1.4.254:33901)
0924 23:18:45.243553 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:45.243562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:45.243587 (+ 25us) server_negotiation.cc:407] Connection header received
0924 23:18:45.244121 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:45.244128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:45.244177 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:45.244296 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:45.245002 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.245938 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.246948 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.247460 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.247560 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:45.250044 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:45.250049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:45.250049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:45.250061 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:45.252147 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:45.252541 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:45.252544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:45.252544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:45.252643 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:45.252846 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:45.252846 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:45.252847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:45.253192 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:45.253351 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:45.254069 (+ 718us) server_negotiation.cc:299] Negotiation successful
0924 23:18:45.254201 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260924 23:18:45.254514 16464 catalog_manager.cc:1347] Generated new cluster ID: f12ad3691aac4f0da76d3feae0e141ea
I20260924 23:18:45.254549 16464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:18:45.277611 16464 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:18:45.278182 16464 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:18:45.284286 16464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Generated new TSK 0
I20260924 23:18:45.284466 16464 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:18:45.319317 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33901
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:45.402299 16468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:45.402501 16468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:45.402523 16468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:45.403764 16468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:45.403810 16468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:45.403837 16468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:18:45.405391 16468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:45.405643 16468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:45.405859 16468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:18:45.410590 16468 server_base.cc:1034] running on GCE node
W20260924 23:18:45.410604 16473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:45.410686 16474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:45.410790 16476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:45.411095 16468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:45.411353 16468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:45.412555 16468 hybrid_clock.cc:648] HybridClock initialized: now 1790291925412537 us; error 31 us; skew 500 ppm
Sep 24 23:18:45 dist-test-slave-s1hw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291925, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:45.414985 16468 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:18:45.415436 16468 webserver.cc:460] Webserver started at http://127.1.4.193:46709/ using document root <none> and password file <none>
I20260924 23:18:45.415642 16468 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:45.415688 16468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:45.415807 16468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:18:45.416816 16468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "0fa3ec975b5d4375b0471353b6ede0c6"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "35a84eae747f2e56d6f5034ef1fc5038"
server_key_iv: "6f094ef528364d8e72d8481970159585"
server_key_version: "encryptionkey@0"
I20260924 23:18:45.417140 16468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "0fa3ec975b5d4375b0471353b6ede0c6"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "35a84eae747f2e56d6f5034ef1fc5038"
server_key_iv: "6f094ef528364d8e72d8481970159585"
server_key_version: "encryptionkey@0"
I20260924 23:18:45.418354 16468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:18:45.419162 16483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:45.419318 16468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:18:45.419435 16468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "0fa3ec975b5d4375b0471353b6ede0c6"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "35a84eae747f2e56d6f5034ef1fc5038"
server_key_iv: "6f094ef528364d8e72d8481970159585"
server_key_version: "encryptionkey@0"
I20260924 23:18:45.419507 16468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:45.449812 16468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:45.452137 16468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:45.452289 16468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:45.452525 16468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:45.452884 16468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:18:45.452926 16468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:45.452960 16468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:18:45.452986 16468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:45.459167 16468 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:45855
I20260924 23:18:45.459208 16596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:45855 every 8 connection(s)
I20260924 23:18:45.459571 16468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:18:45.464633 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 16468
I20260924 23:18:45.464819 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:18:45.465023 1043 external_mini_cluster.cc:1428] Setting key 1f8264845e55047cfcdf2964dbd67a12
Sep 24 23:18:45 dist-test-slave-s1hw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291925, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:45.469062 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:45.460334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:54351 (local address 127.1.4.254:33901)
0924 23:18:45.460391 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:45.460393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:45.460612 (+ 219us) server_negotiation.cc:407] Connection header received
0924 23:18:45.461071 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:45.461073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:45.461085 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:45.461147 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:45.462094 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.462610 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.463689 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.463917 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.463982 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:45.466233 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:45.466235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:45.466235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:45.466242 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:45.467696 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:45.468092 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:45.468093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:45.468093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:45.468154 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:45.468407 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:45.468408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:45.468409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:45.468535 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:45.468595 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:45.468866 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 23:18:45.468957 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:18:45.469219 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:45.460293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:54351)
0924 23:18:45.460458 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:45.460499 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:45.460985 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:45.461206 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:45.461210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:45.461773 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:45.461981 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:45.461987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.462696 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.462697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:45.463588 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:45.463590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.463992 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.463993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:45.464096 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:45.464626 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:45.464635 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:45.466135 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:45.467786 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:45.467793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:45.467794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:45.468021 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:45.468208 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:45.468208 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:45.468209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:45.468328 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:45.468595 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:45.468602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:45.468747 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 23:18:45.468835 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 23:18:45.468997 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260924 23:18:45.469961 16597 heartbeater.cc:347] Connected to a master server at 127.1.4.254:33901
I20260924 23:18:45.470054 16597 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:45.470240 16597 heartbeater.cc:511] Master 127.1.4.254:33901 requested a full tablet report, sending...
I20260924 23:18:45.470790 16406 ts_manager.cc:194] Registered new tserver with Master: 0fa3ec975b5d4375b0471353b6ede0c6 (127.1.4.193:45855)
I20260924 23:18:45.471412 16406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:54351
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:18:45.491315 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33901
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:45.570251 16602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:45.570488 16602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:45.570513 16602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:45.571697 16602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:45.571738 16602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:45.571762 16602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:18:45.573225 16602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:45.573455 16602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:45.573673 16602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:45.578608 16608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:45.578615 16610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:45.578634 16607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:45.578657 16602 server_base.cc:1034] running on GCE node
I20260924 23:18:45.579046 16602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:45.579285 16602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:45.580425 16602 hybrid_clock.cc:648] HybridClock initialized: now 1790291925580415 us; error 32 us; skew 500 ppm
Sep 24 23:18:45 dist-test-slave-s1hw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291925, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:45.582847 16602 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:18:45.583276 16602 webserver.cc:460] Webserver started at http://127.1.4.194:42849/ using document root <none> and password file <none>
I20260924 23:18:45.583525 16602 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:45.583576 16602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:45.583698 16602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:18:45.584708 16602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "d6c3a74839d3424bbba05b9338eedfab"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "ffab8a0e44cfdbb89aa08ac0f48322d0"
server_key_iv: "0381004402c971fa1cdf77cebbd43519"
server_key_version: "encryptionkey@0"
I20260924 23:18:45.585038 16602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "d6c3a74839d3424bbba05b9338eedfab"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "ffab8a0e44cfdbb89aa08ac0f48322d0"
server_key_iv: "0381004402c971fa1cdf77cebbd43519"
server_key_version: "encryptionkey@0"
I20260924 23:18:45.586361 16602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:18:45.587113 16617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:45.587282 16602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:18:45.587389 16602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "d6c3a74839d3424bbba05b9338eedfab"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "ffab8a0e44cfdbb89aa08ac0f48322d0"
server_key_iv: "0381004402c971fa1cdf77cebbd43519"
server_key_version: "encryptionkey@0"
I20260924 23:18:45.587471 16602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:45.611544 16602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:45.613691 16602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:45.613835 16602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:45.614070 16602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:45.614362 16602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:18:45.614393 16602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:45.614431 16602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:18:45.614456 16602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:45.619956 16602 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:44769
I20260924 23:18:45.620007 16730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:44769 every 8 connection(s)
I20260924 23:18:45.620294 16602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:18:45.626101 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 16602
Sep 24 23:18:45 dist-test-slave-s1hw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291925, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:45.626183 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:18:45.626364 1043 external_mini_cluster.cc:1428] Setting key d581a0246ee5f192b08aa0eadea908fa
I20260924 23:18:45.629467 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:45.620943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:36055 (local address 127.1.4.254:33901)
0924 23:18:45.620999 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:45.621001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:45.621172 (+ 171us) server_negotiation.cc:407] Connection header received
0924 23:18:45.621664 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:45.621665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:45.621674 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:45.621716 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:45.622734 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.623380 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.624339 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.624521 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.624577 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:45.626716 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:45.626718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:45.626718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:45.626726 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:45.628108 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:45.628515 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:45.628517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:45.628517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:45.628576 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:45.628822 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:45.628823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:45.628823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:45.628953 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:45.629024 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:45.629290 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 23:18:45.629372 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 23:18:45.629544 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:45.620890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:36055)
0924 23:18:45.621054 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:45.621095 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:45.621566 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:45.621782 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:45.621785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:45.622430 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:45.622639 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:45.622645 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.623468 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.623469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:45.624277 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:45.624279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.624579 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.624580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:45.624674 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:45.625087 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:45.625096 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:45.626618 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:45.628194 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:45.628202 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:45.628204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:45.628441 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:45.628636 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:45.628637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:45.628637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:45.628762 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:45.629029 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:45.629035 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:45.629160 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 23:18:45.629250 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 23:18:45.629369 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260924 23:18:45.630236 16731 heartbeater.cc:347] Connected to a master server at 127.1.4.254:33901
I20260924 23:18:45.630347 16731 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:45.630563 16731 heartbeater.cc:511] Master 127.1.4.254:33901 requested a full tablet report, sending...
I20260924 23:18:45.630934 16406 ts_manager.cc:194] Registered new tserver with Master: d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769)
I20260924 23:18:45.631289 16406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:36055
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:18:45.657655 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33901
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:18:45.737253 16736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:18:45.737461 16736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:18:45.737484 16736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:18:45.738659 16736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:18:45.738701 16736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:18:45.738729 16736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:18:45.740212 16736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:18:45.740440 16736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:18:45.740655 16736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:18:45.745635 16744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:45.745636 16742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:18:45.745631 16741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:18:45.745818 16736 server_base.cc:1034] running on GCE node
I20260924 23:18:45.746078 16736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:18:45.746335 16736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:18:45.747481 16736 hybrid_clock.cc:648] HybridClock initialized: now 1790291925747466 us; error 27 us; skew 500 ppm
Sep 24 23:18:45 dist-test-slave-s1hw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291925, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:18:45.750054 16736 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:18:45.750554 16736 webserver.cc:460] Webserver started at http://127.1.4.195:43747/ using document root <none> and password file <none>
I20260924 23:18:45.750784 16736 fs_manager.cc:359] Metadata directory not provided
I20260924 23:18:45.750833 16736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:18:45.750962 16736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:18:45.752171 16736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "46fc4161b7144764a9fa0c9e55a9b209"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "c2b943f737541b166436f324181e82b5"
server_key_iv: "6edbd7631ab85d93f273e40f11a0652b"
server_key_version: "encryptionkey@0"
I20260924 23:18:45.752535 16736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "46fc4161b7144764a9fa0c9e55a9b209"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "c2b943f737541b166436f324181e82b5"
server_key_iv: "6edbd7631ab85d93f273e40f11a0652b"
server_key_version: "encryptionkey@0"
I20260924 23:18:45.753923 16736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:18:45.754690 16751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:45.754866 16736 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:18:45.754952 16736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "46fc4161b7144764a9fa0c9e55a9b209"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "c2b943f737541b166436f324181e82b5"
server_key_iv: "6edbd7631ab85d93f273e40f11a0652b"
server_key_version: "encryptionkey@0"
I20260924 23:18:45.755025 16736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:18:45.764612 16736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:18:45.766492 16736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:18:45.766628 16736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:18:45.766855 16736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:18:45.767148 16736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:18:45.767181 16736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:45.767210 16736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:18:45.767226 16736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:18:45.772768 16736 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:39101
I20260924 23:18:45.772817 16864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:39101 every 8 connection(s)
I20260924 23:18:45.773125 16736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
Sep 24 23:18:45 dist-test-slave-s1hw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291925, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:18:45.781641 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:45.773858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:50069 (local address 127.1.4.254:33901)
0924 23:18:45.773917 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:45.773919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:45.774107 (+ 188us) server_negotiation.cc:407] Connection header received
0924 23:18:45.774572 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:45.774573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:45.774583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:45.774643 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:45.775617 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.776166 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.777162 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.777361 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.777409 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:45.779424 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:45.779425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:45.779426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:45.779435 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:45.780558 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:45.780942 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:45.780943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:45.780943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:45.780988 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:45.781168 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:45.781169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:45.781169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:45.781265 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:45.781309 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:45.781517 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 23:18:45.781574 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 23:18:45.781817 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:45.773838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:50069)
0924 23:18:45.773989 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:45.774030 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:45.774490 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:45.774701 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:45.774704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:45.775271 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:18:45.775519 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:45.775526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.776273 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.776274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:45.777082 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:45.777084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.777421 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.777422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:45.777518 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:45.777954 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:18:45.777962 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:18:45.779311 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:18:45.780640 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:45.780646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:45.780648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:45.780875 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:45.781035 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:18:45.781036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:18:45.781036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:18:45.781127 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:18:45.781308 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:18:45.781311 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:18:45.781422 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 23:18:45.781500 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 23:18:45.781615 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260924 23:18:45.782146 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 16736
I20260924 23:18:45.782250 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:18:45.782320 16865 heartbeater.cc:347] Connected to a master server at 127.1.4.254:33901
I20260924 23:18:45.782414 16865 heartbeater.cc:464] Registering TS with master...
I20260924 23:18:45.782447 1043 external_mini_cluster.cc:1428] Setting key e89369dd1d7e313c4e1cd90e3234a89f
I20260924 23:18:45.782598 16865 heartbeater.cc:511] Master 127.1.4.254:33901 requested a full tablet report, sending...
I20260924 23:18:45.783077 16405 ts_manager.cc:194] Registered new tserver with Master: 46fc4161b7144764a9fa0c9e55a9b209 (127.1.4.195:39101)
I20260924 23:18:45.783543 16405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:50069
I20260924 23:18:45.794078 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:18:45.800823 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:45.795292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.1.4.254:33901)
0924 23:18:45.795337 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:45.795339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:45.795470 (+ 131us) server_negotiation.cc:407] Connection header received
0924 23:18:45.795542 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:45.795542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:45.795554 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:45.795598 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:18:45.796140 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.796627 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.797333 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.797513 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.797564 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:45.798678 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:18:45.798679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:18:45.798679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:18:45.798687 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:18:45.799800 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:45.800100 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:45.800101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:45.800102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:45.800150 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:18:45.800314 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:18:45.800315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:18:45.800316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:18:45.800459 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:18:45.800548 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:45.800649 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 23:18:45.800728 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:18:45.802368 16405 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:18:45.803985 16405 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:18:45.809966 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:45.806276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:45855 (local address 127.0.0.1:46972)
0924 23:18:45.806454 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:45.806498 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:45.806568 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:45.806903 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:45.806906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:45.806908 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:45.806977 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:45.806979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.808040 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.808041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:45.808993 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:45.808995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.809650 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.809651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:45.809744 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:45.809761 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 23:18:45.809830 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 23:18:45.809882 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
I20260924 23:18:45.809981 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:45.806449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46972 (local address 127.1.4.193:45855)
0924 23:18:45.806669 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:45.806674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:45.806683 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:18:45.806722 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:45.806724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:45.806739 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:45.806812 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:45.807054 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.807948 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.809066 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.809570 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.809641 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:45.809705 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:45.809849 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 23:18:45.809890 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":106,"threads_started":1}
I20260924 23:18:45.810621 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:45.806475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:44769 (local address 127.0.0.1:39310)
0924 23:18:45.806761 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:45.806772 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:45.806821 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:45.807179 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:45.807180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:45.807182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:45.807237 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:45.807239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.808666 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.808667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:45.809526 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:45.809528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.810366 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.810367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:45.810461 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:45.810465 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:18:45.810509 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:18:45.810550 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":12,"thread_start_us":82,"threads_started":1}
I20260924 23:18:45.810640 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:45.806598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.1.4.194:44769)
0924 23:18:45.806970 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:45.806974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:45.806983 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:18:45.807019 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:45.807021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:45.807036 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:45.807104 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:45.807429 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.808557 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.809612 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.810289 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.810353 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:45.810415 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:45.810534 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 23:18:45.810569 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":77,"threads_started":1}
I20260924 23:18:45.811043 16531 tablet_service.cc:1467] Processing CreateTablet for tablet b1fbf058e0ca45568a4f805d6d9bc3a5 (DEFAULT_TABLE table=test-table [id=0cab5c5ee34e48bc8040f7829cce63fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:18:45.811455 16531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fbf058e0ca45568a4f805d6d9bc3a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:45.811486 16665 tablet_service.cc:1467] Processing CreateTablet for tablet b1fbf058e0ca45568a4f805d6d9bc3a5 (DEFAULT_TABLE table=test-table [id=0cab5c5ee34e48bc8040f7829cce63fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:18:45.811805 16665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fbf058e0ca45568a4f805d6d9bc3a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:45.811901 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:45.806634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:39101 (local address 127.0.0.1:50918)
0924 23:18:45.806990 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:45.807000 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:45.807039 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:45.807309 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:45.807310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:45.807312 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:45.807395 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:45.807397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.809599 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.809599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:45.810704 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:45.810707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.811673 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.811674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:45.811746 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:45.811750 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:18:45.811798 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 23:18:45.811835 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":1,"thread_start_us":52,"threads_started":1}
I20260924 23:18:45.811936 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:45.806811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50918 (local address 127.1.4.195:39101)
0924 23:18:45.807173 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:45.807178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:45.807186 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:18:45.807219 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:45.807220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:45.807231 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:45.807293 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:45.807481 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.808557 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.810780 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:45.811526 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:45.811589 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:45.811641 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:45.811821 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 23:18:45.811855 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":94,"threads_started":1}
I20260924 23:18:45.814802 16883 tablet_bootstrap.cc:492] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab: Bootstrap starting.
I20260924 23:18:45.815524 16882 tablet_bootstrap.cc:492] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: Bootstrap starting.
I20260924 23:18:45.815553 16883 tablet_bootstrap.cc:654] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:45.815513 16799 tablet_service.cc:1467] Processing CreateTablet for tablet b1fbf058e0ca45568a4f805d6d9bc3a5 (DEFAULT_TABLE table=test-table [id=0cab5c5ee34e48bc8040f7829cce63fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:18:45.815855 16799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fbf058e0ca45568a4f805d6d9bc3a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:18:45.816076 16882 tablet_bootstrap.cc:654] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:45.816109 16883 log.cc:824] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:45.816556 16882 log.cc:824] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:45.816902 16883 tablet_bootstrap.cc:492] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab: No bootstrap required, opened a new log
I20260924 23:18:45.816963 16883 ts_tablet_manager.cc:1397] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:18:45.817174 16882 tablet_bootstrap.cc:492] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: No bootstrap required, opened a new log
I20260924 23:18:45.817220 16882 ts_tablet_manager.cc:1397] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:18:45.818616 16882 raft_consensus.cc:348] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } } peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } }
I20260924 23:18:45.818763 16882 raft_consensus.cc:374] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:45.818783 16882 raft_consensus.cc:729] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa3ec975b5d4375b0471353b6ede0c6, State: Initialized, Role: FOLLOWER
I20260924 23:18:45.818877 16882 consensus_queue.cc:260] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } } peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } }
I20260924 23:18:45.818897 16883 raft_consensus.cc:348] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } } peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } }
I20260924 23:18:45.819064 16883 raft_consensus.cc:374] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:45.819097 16883 raft_consensus.cc:729] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c3a74839d3424bbba05b9338eedfab, State: Initialized, Role: FOLLOWER
I20260924 23:18:45.819103 16886 tablet_bootstrap.cc:492] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209: Bootstrap starting.
I20260924 23:18:45.819223 16883 consensus_queue.cc:260] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } } peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } }
I20260924 23:18:45.819294 16882 ts_tablet_manager.cc:1428] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:18:45.819473 16597 heartbeater.cc:503] Master 127.1.4.254:33901 was elected leader, sending a full tablet report...
I20260924 23:18:45.819523 16883 ts_tablet_manager.cc:1428] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 23:18:45.819639 16731 heartbeater.cc:503] Master 127.1.4.254:33901 was elected leader, sending a full tablet report...
I20260924 23:18:45.819800 16886 tablet_bootstrap.cc:654] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209: Neither blocks nor log segments found. Creating new log.
I20260924 23:18:45.820322 16886 log.cc:824] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209: Log is configured to *not* fsync() on all Append() calls
I20260924 23:18:45.821064 16886 tablet_bootstrap.cc:492] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209: No bootstrap required, opened a new log
I20260924 23:18:45.821118 16886 ts_tablet_manager.cc:1397] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:18:45.822582 16886 raft_consensus.cc:348] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } } peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } }
I20260924 23:18:45.822739 16886 raft_consensus.cc:374] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:18:45.822798 16886 raft_consensus.cc:729] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fc4161b7144764a9fa0c9e55a9b209, State: Initialized, Role: FOLLOWER
I20260924 23:18:45.822913 16886 consensus_queue.cc:260] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } } peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } }
I20260924 23:18:45.823164 16886 ts_tablet_manager.cc:1428] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:18:45.823197 16865 heartbeater.cc:503] Master 127.1.4.254:33901 was elected leader, sending a full tablet report...
W20260924 23:18:45.870839 16732 tablet.cc:2369] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:18:45.960369 16598 tablet.cc:2369] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:18:46.023667 16866 tablet.cc:2369] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:18:46.049942 16890 raft_consensus.cc:482] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:18:46.050066 16890 raft_consensus.cc:504] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } } peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } }
I20260924 23:18:46.050437 16890 leader_election.cc:288] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fa3ec975b5d4375b0471353b6ede0c6 (127.1.4.193:45855), d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769)
I20260924 23:18:46.053558 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:46.050887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:45755 (local address 127.1.4.194:44769)
0924 23:18:46.050932 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:46.050934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:46.050997 (+ 63us) server_negotiation.cc:407] Connection header received
0924 23:18:46.051080 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:46.051081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:46.051090 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:46.051141 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:46.051299 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:46.051893 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:46.052872 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:46.053248 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:46.053308 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:46.053319 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:46.053451 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 23:18:46.053487 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:18:46.053568 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:46.050695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:35599 (local address 127.1.4.193:45855)
0924 23:18:46.050760 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:46.050761 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:46.050773 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:18:46.050806 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:46.050807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:46.050819 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:46.050895 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:18:46.051054 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:46.051697 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:46.052867 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:46.053235 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:46.053290 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:46.053301 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:46.053464 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 23:18:46.053501 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 23:18:46.053579 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:46.050618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:45855 (local address 127.1.4.195:35599)
0924 23:18:46.050671 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:46.050684 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:46.050727 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:46.050917 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:46.050918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:46.050920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:46.050990 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:46.050992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:46.051779 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:46.051780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:46.052776 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:46.052778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:46.053312 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:46.053313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:46.053391 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:46.053395 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:18:46.053443 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 23:18:46.053487 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":2}
I20260924 23:18:46.053587 16891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:46.050740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:44769 (local address 127.1.4.195:45755)
0924 23:18:46.050913 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:46.050923 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:18:46.051024 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:18:46.051154 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:18:46.051155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:18:46.051156 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:18:46.051235 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:46.051236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:46.051970 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:46.051971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:46.052789 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:18:46.052792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:18:46.053326 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:18:46.053327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:18:46.053386 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:46.053390 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:18:46.053434 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:18:46.053472 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":102,"threads_started":1}
I20260924 23:18:46.053939 16551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fbf058e0ca45568a4f805d6d9bc3a5" candidate_uuid: "46fc4161b7144764a9fa0c9e55a9b209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" is_pre_election: true
I20260924 23:18:46.053979 16685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fbf058e0ca45568a4f805d6d9bc3a5" candidate_uuid: "46fc4161b7144764a9fa0c9e55a9b209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c3a74839d3424bbba05b9338eedfab" is_pre_election: true
I20260924 23:18:46.054090 16551 raft_consensus.cc:2454] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46fc4161b7144764a9fa0c9e55a9b209 in term 0.
I20260924 23:18:46.054107 16685 raft_consensus.cc:2454] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46fc4161b7144764a9fa0c9e55a9b209 in term 0.
I20260924 23:18:46.054332 16753 leader_election.cc:302] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa3ec975b5d4375b0471353b6ede0c6, 46fc4161b7144764a9fa0c9e55a9b209; no voters:
I20260924 23:18:46.054486 16890 raft_consensus.cc:2781] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:18:46.054558 16890 raft_consensus.cc:482] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:18:46.054574 16890 raft_consensus.cc:3037] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:46.055395 16890 raft_consensus.cc:504] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } } peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } }
I20260924 23:18:46.055507 16890 leader_election.cc:288] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [CANDIDATE]: Term 1 election: Requested vote from peers 0fa3ec975b5d4375b0471353b6ede0c6 (127.1.4.193:45855), d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769)
I20260924 23:18:46.055691 16685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fbf058e0ca45568a4f805d6d9bc3a5" candidate_uuid: "46fc4161b7144764a9fa0c9e55a9b209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c3a74839d3424bbba05b9338eedfab"
I20260924 23:18:46.055704 16551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fbf058e0ca45568a4f805d6d9bc3a5" candidate_uuid: "46fc4161b7144764a9fa0c9e55a9b209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fa3ec975b5d4375b0471353b6ede0c6"
I20260924 23:18:46.055775 16685 raft_consensus.cc:3037] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:46.055776 16551 raft_consensus.cc:3037] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:18:46.056604 16551 raft_consensus.cc:2454] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fc4161b7144764a9fa0c9e55a9b209 in term 1.
I20260924 23:18:46.056604 16685 raft_consensus.cc:2454] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fc4161b7144764a9fa0c9e55a9b209 in term 1.
I20260924 23:18:46.056782 16753 leader_election.cc:302] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa3ec975b5d4375b0471353b6ede0c6, 46fc4161b7144764a9fa0c9e55a9b209; no voters:
I20260924 23:18:46.056864 16890 raft_consensus.cc:2781] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:18:46.057004 16890 raft_consensus.cc:686] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [term 1 LEADER]: Becoming Leader. State: Replica: 46fc4161b7144764a9fa0c9e55a9b209, State: Running, Role: LEADER
I20260924 23:18:46.057082 16890 consensus_queue.cc:237] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } } peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } }
I20260924 23:18:46.057876 16406 catalog_manager.cc:5495] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46fc4161b7144764a9fa0c9e55a9b209 (127.1.4.195). New cstate: current_term: 1 leader_uuid: "46fc4161b7144764a9fa0c9e55a9b209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } health_report { overall_health: HEALTHY } } }
I20260924 23:18:46.092109 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:46.089776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50926 (local address 127.1.4.195:39101)
0924 23:18:46.089816 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:18:46.089818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:18:46.089848 (+ 30us) server_negotiation.cc:407] Connection header received
0924 23:18:46.089926 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:18:46.089927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:18:46.089948 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:18:46.090000 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:18:46.090124 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:46.090632 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:46.091455 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:18:46.091638 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:18:46.091683 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:18:46.091855 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:18:46.091963 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:18:46.092020 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:18:46.092034 (+ 14us) server_negotiation.cc:299] Negotiation successful
0924 23:18:46.092050 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:18:46.093892 16551 raft_consensus.cc:1261] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Refusing update from remote peer 46fc4161b7144764a9fa0c9e55a9b209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:18:46.093892 16685 raft_consensus.cc:1261] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab [term 1 FOLLOWER]: Refusing update from remote peer 46fc4161b7144764a9fa0c9e55a9b209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:18:46.094254 16890 consensus_queue.cc:1035] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:18:46.094389 16892 consensus_queue.cc:1035] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:18:46.096944 16896 mvcc.cc:205] Tried to move back new op lower bound from 7333035729277906944 to 7333035729130278912. Current Snapshot: MvccSnapshot[applied={T|T < 7333035729277906944 or (T in {7333035729277906944})}]
I20260924 23:18:46.099200 16405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:18:46.099287 16405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:18:46.099980 16405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Sending DeleteTablet for 3 replicas of tablet b1fbf058e0ca45568a4f805d6d9bc3a5
I20260924 23:18:46.100287 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 16468
I20260924 23:18:46.100368 16531 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fbf058e0ca45568a4f805d6d9bc3a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:18:46 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:46972
I20260924 23:18:46.100752 16665 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fbf058e0ca45568a4f805d6d9bc3a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:18:46 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:39310
I20260924 23:18:46.100824 16799 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fbf058e0ca45568a4f805d6d9bc3a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:18:46 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:50918
I20260924 23:18:46.101014 16904 tablet_replica.cc:330] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab: stopping tablet replica
I20260924 23:18:46.101119 16905 tablet_replica.cc:330] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209: stopping tablet replica
I20260924 23:18:46.101120 16904 raft_consensus.cc:2229] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:18:46.101218 16904 raft_consensus.cc:2258] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:18:46.101300 16905 raft_consensus.cc:2229] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:18:46.101447 16905 raft_consensus.cc:2258] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:18:46.101701 16904 ts_tablet_manager.cc:1905] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:18:46.101876 16905 ts_tablet_manager.cc:1905] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:18:46.103888 16905 ts_tablet_manager.cc:1918] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:18:46.103925 16904 ts_tablet_manager.cc:1918] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:18:46.104022 16905 log.cc:1196] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/b1fbf058e0ca45568a4f805d6d9bc3a5
I20260924 23:18:46.104058 16904 log.cc:1196] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/b1fbf058e0ca45568a4f805d6d9bc3a5
I20260924 23:18:46.104328 16905 ts_tablet_manager.cc:1939] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 46fc4161b7144764a9fa0c9e55a9b209: Deleting consensus metadata
I20260924 23:18:46.104552 16904 ts_tablet_manager.cc:1939] T b1fbf058e0ca45568a4f805d6d9bc3a5 P d6c3a74839d3424bbba05b9338eedfab: Deleting consensus metadata
I20260924 23:18:46.104729 16391 catalog_manager.cc:4841] TS 46fc4161b7144764a9fa0c9e55a9b209 (127.1.4.195:39101): tablet b1fbf058e0ca45568a4f805d6d9bc3a5 (table test-table [id=0cab5c5ee34e48bc8040f7829cce63fb]) successfully deleted
I20260924 23:18:46.105144 16391 catalog_manager.cc:4841] TS d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769): tablet b1fbf058e0ca45568a4f805d6d9bc3a5 (table test-table [id=0cab5c5ee34e48bc8040f7829cce63fb]) successfully deleted
W20260924 23:18:46.106727 16391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.4.193:45855 (error 108)
I20260924 23:18:46.106868 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 16602
I20260924 23:18:46.110865 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:18:46.110687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:45855 (local address 127.0.0.1:46982)
0924 23:18:46.110776 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 23:18:46.110828 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:45855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 23:18:46.111158 16391 catalog_manager.cc:4584] TS 0fa3ec975b5d4375b0471353b6ede0c6 (127.1.4.193:45855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b1fbf058e0ca45568a4f805d6d9bc3a5: Network error: Client connection negotiation failed: client connection to 127.1.4.193:45855: connect: Connection refused (error 111)
I20260924 23:18:46.111294 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 16736
I20260924 23:18:46.116099 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 16374
I20260924 23:19:06.120776 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:33901
--webserver_interface=127.1.4.254
--webserver_port=42041
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:33901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:06.201862 16907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:06.202075 16907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:06.202097 16907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:06.203291 16907 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 23:19:06.203334 16907 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 23:19:06.203346 16907 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 23:19:06.203356 16907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:19:06.203397 16907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:06.203414 16907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:06.203437 16907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:19:06.203467 16907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:19:06.204998 16907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:33901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:33901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=42041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:06.205221 16907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:06.205452 16907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:06.214028 16913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:06.214161 16916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:06.214195 16907 server_base.cc:1034] running on GCE node
W20260924 23:19:06.214043 16914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:06.214530 16907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:06.214861 16907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:06.216005 16907 hybrid_clock.cc:648] HybridClock initialized: now 1790291946215987 us; error 31 us; skew 500 ppm
Sep 24 23:19:06 dist-test-slave-s1hw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291946, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:06.221938 16907 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:19:06.222446 16907 webserver.cc:460] Webserver started at http://127.1.4.254:42041/ using document root <none> and password file <none>
I20260924 23:19:06.222681 16907 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:06.222738 16907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:06.224185 16907 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:19:06.224867 16923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:06.225044 16907 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:19:06.225131 16907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "67ac7572fd034c6eae1337ee7ae185cf"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "61c410df8ed17178767ab6a3352d484f"
server_key_iv: "59dcb1fd36487ba29d48dc703e029ac4"
server_key_version: "encryptionkey@0"
I20260924 23:19:06.225543 16907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:06.238075 16907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:06.240118 16907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:06.240231 16907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:06.244017 16907 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:33901
I20260924 23:19:06.244060 16975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:33901 every 8 connection(s)
I20260924 23:19:06.244422 16907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:19:06.245841 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 16907
I20260924 23:19:06.245918 16976 sys_catalog.cc:263] Verifying existing consensus state
I20260924 23:19:06.246120 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:45855
--local_ip_for_outbound_sockets=127.1.4.193
--tserver_master_addrs=127.1.4.254:33901
--webserver_port=46709
--webserver_interface=127.1.4.193
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:19:06.246644 16976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Bootstrap starting.
I20260924 23:19:06.254773 16976 log.cc:824] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:06.256734 16976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:19:06.256994 16976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Bootstrap complete.
I20260924 23:19:06.259078 16976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } }
I20260924 23:19:06.259322 16976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67ac7572fd034c6eae1337ee7ae185cf, State: Initialized, Role: FOLLOWER
I20260924 23:19:06.259505 16976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } }
I20260924 23:19:06.259584 16976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:19:06.259637 16976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:19:06.259683 16976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 1 FOLLOWER]: Advancing to term 2
I20260924 23:19:06.260669 16976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } }
I20260924 23:19:06.260774 16976 leader_election.cc:302] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67ac7572fd034c6eae1337ee7ae185cf; no voters:
I20260924 23:19:06.260946 16976 leader_election.cc:288] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 23:19:06.260993 16980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 2 FOLLOWER]: Leader election won for term 2
I20260924 23:19:06.261145 16980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 2 LEADER]: Becoming Leader. State: Replica: 67ac7572fd034c6eae1337ee7ae185cf, State: Running, Role: LEADER
I20260924 23:19:06.261214 16976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:19:06.261292 16980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } }
I20260924 23:19:06.261696 16980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67ac7572fd034c6eae1337ee7ae185cf. Latest consensus state: current_term: 2 leader_uuid: "67ac7572fd034c6eae1337ee7ae185cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } } }
I20260924 23:19:06.261803 16980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:06.262079 16981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "67ac7572fd034c6eae1337ee7ae185cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } } }
I20260924 23:19:06.262212 16981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:06.262370 16989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:19:06.263139 16989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:19:06.263252 16989 catalog_manager.cc:1259] Loaded cluster ID: f12ad3691aac4f0da76d3feae0e141ea
I20260924 23:19:06.263525 16989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:19:06.264719 16989 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:19:06.264894 16989 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Loaded TSK: 0
I20260924 23:19:06.287150 16989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Generated new TSK 1
I20260924 23:19:06.287281 16989 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 23:19:06.334566 16978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:06.334903 16978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:06.334929 16978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:06.336331 16978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:06.336380 16978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:06.336408 16978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:19:06.337985 16978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:45855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=46709
--tserver_master_addrs=127.1.4.254:33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:06.338217 16978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:06.338451 16978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:06.343415 17004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:06.343514 16978 server_base.cc:1034] running on GCE node
W20260924 23:19:06.343446 17006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:06.343587 17003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:06.343807 16978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:06.344044 16978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:06.345228 16978 hybrid_clock.cc:648] HybridClock initialized: now 1790291946345196 us; error 51 us; skew 500 ppm
Sep 24 23:19:06 dist-test-slave-s1hw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291946, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:06.348107 16978 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:19:06.348670 16978 webserver.cc:460] Webserver started at http://127.1.4.193:46709/ using document root <none> and password file <none>
I20260924 23:19:06.348930 16978 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:06.349006 16978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:06.350773 16978 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:19:06.351590 17013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:06.351743 16978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:06.351835 16978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "0fa3ec975b5d4375b0471353b6ede0c6"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "35a84eae747f2e56d6f5034ef1fc5038"
server_key_iv: "6f094ef528364d8e72d8481970159585"
server_key_version: "encryptionkey@0"
I20260924 23:19:06.352324 16978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:06.366664 16978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:06.369184 16978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:06.369352 16978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:06.369616 16978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:06.370143 17020 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 23:19:06.371773 16978 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 23:19:06.371836 16978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 23:19:06.371873 16978 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 23:19:06.372779 16978 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 23:19:06.372825 16978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 23:19:06.372906 17020 tablet_bootstrap.cc:492] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: Bootstrap starting.
I20260924 23:19:06.379767 16978 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:45855
I20260924 23:19:06.380009 17127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:45855 every 8 connection(s)
I20260924 23:19:06.380303 16978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:19:06.380888 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 16978
I20260924 23:19:06.381168 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:44769
--local_ip_for_outbound_sockets=127.1.4.194
--tserver_master_addrs=127.1.4.254:33901
--webserver_port=42849
--webserver_interface=127.1.4.194
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:19:06.383253 17020 log.cc:824] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:06.385385 17020 tablet_bootstrap.cc:492] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:19:06.385637 17020 tablet_bootstrap.cc:492] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: Bootstrap complete.
I20260924 23:19:06.386705 17020 ts_tablet_manager.cc:1397] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260924 23:19:06.388816 17020 raft_consensus.cc:348] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } } peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } }
I20260924 23:19:06.389240 17020 raft_consensus.cc:729] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa3ec975b5d4375b0471353b6ede0c6, State: Initialized, Role: FOLLOWER
I20260924 23:19:06.389459 17020 consensus_queue.cc:260] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } } peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } }
I20260924 23:19:06.389808 17020 ts_tablet_manager.cc:1428] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
Sep 24 23:19:06 dist-test-slave-s1hw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291946, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:19:06.399821 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.381053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:55699)
0924 23:19:06.383545 (+ 2492us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:06.383602 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:06.388357 (+ 4755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:06.389010 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:06.389014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:06.389421 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:06.389690 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:06.389696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.391358 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.391358 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:06.392648 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:06.392651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.393450 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.393451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:06.393579 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.394148 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:19:06.394157 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:06.396009 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:06.398024 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:06.398031 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:06.398033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:06.398317 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:06.398579 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:06.398580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:06.398580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:06.398724 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:06.399150 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:06.399155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:06.399310 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 23:19:06.399435 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 23:19:06.399597 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2439,"thread_start_us":103,"threads_started":1}
I20260924 23:19:06.400354 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.381117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:55699 (local address 127.1.4.254:33901)
0924 23:19:06.383667 (+ 2550us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:06.383675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:06.383724 (+ 49us) server_negotiation.cc:407] Connection header received
0924 23:19:06.388642 (+ 4918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:06.388651 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:06.388718 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:06.388822 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:06.389828 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.391209 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.392748 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.393354 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.393474 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.396115 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:06.396120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:06.396120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:06.396135 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:06.397945 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:06.398407 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:06.398410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:06.398411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:06.398520 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:06.398795 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:06.398796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:06.398796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:06.399079 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:06.399157 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:06.400031 (+ 874us) server_negotiation.cc:299] Negotiation successful
0924 23:19:06.400170 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2491,"thread_start_us":87,"threads_started":1}
I20260924 23:19:06.401062 17128 heartbeater.cc:347] Connected to a master server at 127.1.4.254:33901
I20260924 23:19:06.401157 17128 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:06.401337 17128 heartbeater.cc:511] Master 127.1.4.254:33901 requested a full tablet report, sending...
I20260924 23:19:06.402031 16940 ts_manager.cc:194] Registered new tserver with Master: 0fa3ec975b5d4375b0471353b6ede0c6 (127.1.4.193:45855)
I20260924 23:19:06.402993 16940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:55699
I20260924 23:19:06.403867 17128 heartbeater.cc:503] Master 127.1.4.254:33901 was elected leader, sending a full tablet report...
I20260924 23:19:06.407317 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.402689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.1.4.193:45855)
0924 23:19:06.403553 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:06.403557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:06.403569 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:19:06.404124 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:06.404127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:06.404142 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:06.404201 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:06.404352 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.405379 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.406293 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.406942 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.407012 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.407061 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:06.407216 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 23:19:06.407253 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":83,"threads_started":1}
I20260924 23:19:06.407323 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.402616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:45855 (local address 127.0.0.1:39726)
0924 23:19:06.403272 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:06.403316 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:06.403391 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:06.404218 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:06.404221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:06.404223 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:06.404285 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:06.404287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.405455 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.405456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:06.406222 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:06.406224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.407040 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.407041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:06.407111 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.407123 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 23:19:06.407204 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 23:19:06.407264 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":100,"threads_started":1}
I20260924 23:19:06.407739 17062 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fbf058e0ca45568a4f805d6d9bc3a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:18:46 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:39726
I20260924 23:19:06.408259 17137 tablet_replica.cc:330] stopping tablet replica
I20260924 23:19:06.408363 17137 raft_consensus.cc:2229] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:19:06.408444 17137 raft_consensus.cc:2258] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:19:06.408737 17137 ts_tablet_manager.cc:1905] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:19:06.410826 17137 ts_tablet_manager.cc:1918] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:19:06.410887 17137 log.cc:1196] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal/wals/b1fbf058e0ca45568a4f805d6d9bc3a5
I20260924 23:19:06.411063 17137 ts_tablet_manager.cc:1939] T b1fbf058e0ca45568a4f805d6d9bc3a5 P 0fa3ec975b5d4375b0471353b6ede0c6: Deleting consensus metadata
I20260924 23:19:06.411554 16925 catalog_manager.cc:4841] TS 0fa3ec975b5d4375b0471353b6ede0c6 (127.1.4.193:45855): tablet b1fbf058e0ca45568a4f805d6d9bc3a5 (table test-table [id=0cab5c5ee34e48bc8040f7829cce63fb]) successfully deleted
W20260924 23:19:06.469802 17132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:06.470019 17132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:06.470043 17132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:06.471248 17132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:06.471294 17132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:06.471319 17132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:19:06.472796 17132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:44769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=42849
--tserver_master_addrs=127.1.4.254:33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:06.472991 17132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:06.473188 17132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:06.478046 17142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:06.478049 17143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:06.478190 17132 server_base.cc:1034] running on GCE node
W20260924 23:19:06.478122 17145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:06.478497 17132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:06.478717 17132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:06.479849 17132 hybrid_clock.cc:648] HybridClock initialized: now 1790291946479837 us; error 27 us; skew 500 ppm
Sep 24 23:19:06 dist-test-slave-s1hw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291946, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:06.482180 17132 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:19:06.482627 17132 webserver.cc:460] Webserver started at http://127.1.4.194:42849/ using document root <none> and password file <none>
I20260924 23:19:06.482856 17132 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:06.482921 17132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:06.484292 17132 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:06.484866 17152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:06.485010 17132 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:06.485088 17132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "d6c3a74839d3424bbba05b9338eedfab"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "ffab8a0e44cfdbb89aa08ac0f48322d0"
server_key_iv: "0381004402c971fa1cdf77cebbd43519"
server_key_version: "encryptionkey@0"
I20260924 23:19:06.485432 17132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:06.502364 17132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:06.504671 17132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:06.504812 17132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:06.505029 17132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:06.505331 17132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:06.505364 17132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:06.505394 17132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:06.505411 17132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:06.510921 17132 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:44769
I20260924 23:19:06.510991 17265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:44769 every 8 connection(s)
I20260924 23:19:06.511310 17132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:19:06 dist-test-slave-s1hw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291946, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:19:06.518939 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 17132
I20260924 23:19:06.519212 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:39101
--local_ip_for_outbound_sockets=127.1.4.195
--tserver_master_addrs=127.1.4.254:33901
--webserver_port=43747
--webserver_interface=127.1.4.195
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:19:06.520433 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.512035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:58491 (local address 127.1.4.254:33901)
0924 23:19:06.512115 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:06.512118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:06.512320 (+ 202us) server_negotiation.cc:407] Connection header received
0924 23:19:06.512788 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:06.512790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:06.512803 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:06.512847 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:06.513730 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.514248 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.515189 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.515461 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.515515 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.517743 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:06.517745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:06.517745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:06.517752 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:06.518994 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:06.519449 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:06.519451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:06.519451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:06.519510 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:06.519727 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:06.519728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:06.519729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:06.519906 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:06.519966 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:06.520237 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 23:19:06.520338 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 23:19:06.520627 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.512013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:58491)
0924 23:19:06.512191 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:06.512230 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:06.512696 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:06.512909 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:06.512912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:06.513449 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:06.513644 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:06.513649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.514321 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.514322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:06.515115 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:06.515118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.515540 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.515541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:06.515662 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.516256 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:19:06.516265 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:06.517646 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:06.519074 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:06.519082 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:06.519084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:06.519356 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:06.519565 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:06.519566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:06.519566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:06.519671 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:06.519972 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:06.519979 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:06.520117 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 23:19:06.520211 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 23:19:06.520422 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260924 23:19:06.521037 17266 heartbeater.cc:347] Connected to a master server at 127.1.4.254:33901
I20260924 23:19:06.521132 17266 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:06.521311 17266 heartbeater.cc:511] Master 127.1.4.254:33901 requested a full tablet report, sending...
I20260924 23:19:06.521683 16940 ts_manager.cc:194] Registered new tserver with Master: d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769)
I20260924 23:19:06.522058 16940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:58491
W20260924 23:19:06.599207 17269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:06.599453 17269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:06.599488 17269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:06.600660 17269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:06.600705 17269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:06.600741 17269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:19:06.602246 17269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:39101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=43747
--tserver_master_addrs=127.1.4.254:33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:06.602484 17269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:06.602702 17269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:06.607697 17275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:06.607795 17269 server_base.cc:1034] running on GCE node
W20260924 23:19:06.607789 17274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:06.607738 17277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:06.608141 17269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:06.608392 17269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:06.609529 17269 hybrid_clock.cc:648] HybridClock initialized: now 1790291946609518 us; error 28 us; skew 500 ppm
Sep 24 23:19:06 dist-test-slave-s1hw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291946, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:06.612035 17269 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:19:06.612481 17269 webserver.cc:460] Webserver started at http://127.1.4.195:43747/ using document root <none> and password file <none>
I20260924 23:19:06.612679 17269 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:06.612739 17269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:06.614130 17269 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:06.614694 17284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:06.614856 17269 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:06.614936 17269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "46fc4161b7144764a9fa0c9e55a9b209"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "c2b943f737541b166436f324181e82b5"
server_key_iv: "6edbd7631ab85d93f273e40f11a0652b"
server_key_version: "encryptionkey@0"
I20260924 23:19:06.615271 17269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:06.622359 17269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:06.624539 17269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:06.624673 17269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:06.624917 17269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:06.625209 17269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:06.625242 17269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:06.625272 17269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:06.625288 17269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:06.630977 17269 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:39101
I20260924 23:19:06.631021 17397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:39101 every 8 connection(s)
I20260924 23:19:06.631350 17269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:19:06.634028 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 17269
Sep 24 23:19:06 dist-test-slave-s1hw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291946, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:19:06.641700 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.634392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38032 (local address 127.1.4.254:33901)
0924 23:19:06.634489 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:06.634490 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:06.634643 (+ 153us) server_negotiation.cc:407] Connection header received
0924 23:19:06.634678 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:06.634679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:06.634694 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:06.634769 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:06.635482 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.636163 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.637146 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.637337 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.637397 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.638595 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:06.638596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:06.638597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:06.638603 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:06.640543 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:06.640946 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:06.640947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:06.640948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:06.641023 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:06.641213 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:06.641213 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:06.641214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:06.641364 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:06.641527 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:06.641562 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 23:19:06.641625 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":49,"threads_started":1}
I20260924 23:19:06.641886 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.632037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:49495 (local address 127.1.4.254:33901)
0924 23:19:06.632085 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:06.632087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:06.632304 (+ 217us) server_negotiation.cc:407] Connection header received
0924 23:19:06.632775 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:06.632777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:06.632788 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:06.632835 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:06.633719 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.634269 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.635740 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.636042 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.636130 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.638670 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:06.638671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:06.638671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:06.638677 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:06.640568 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:06.640965 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:06.640966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:06.640966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:06.641034 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:06.641302 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:06.641303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:06.641303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:06.641393 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:06.641444 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:06.641749 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 23:19:06.641820 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":21}
I20260924 23:19:06.642066 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.632037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:49495)
0924 23:19:06.632177 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:06.632210 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:06.632685 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:06.632900 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:06.632904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:06.633456 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:06.633647 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:06.633652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.634394 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.634395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:06.635631 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:06.635634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.636127 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.636128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:06.636235 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.636849 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:19:06.636859 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:06.638570 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:06.640632 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:06.640639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:06.640641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:06.640907 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:06.641089 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:06.641089 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:06.641089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:06.641226 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:06.641445 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:06.641450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:06.641603 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 23:19:06.641712 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 23:19:06.641879 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260924 23:19:06.642442 17398 heartbeater.cc:347] Connected to a master server at 127.1.4.254:33901
I20260924 23:19:06.642547 17398 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:06.642727 17398 heartbeater.cc:511] Master 127.1.4.254:33901 requested a full tablet report, sending...
I20260924 23:19:06.643108 16939 ts_manager.cc:194] Registered new tserver with Master: 46fc4161b7144764a9fa0c9e55a9b209 (127.1.4.195:39101)
I20260924 23:19:06.643515 16939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:49495
I20260924 23:19:06.653687 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:19:06.656580 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.654014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38048 (local address 127.1.4.254:33901)
0924 23:19:06.654076 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:06.654078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:06.654159 (+ 81us) server_negotiation.cc:407] Connection header received
0924 23:19:06.654237 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:06.654238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:06.654293 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:06.654356 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:19:06.654483 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.654963 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.655783 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.656001 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.656051 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.656199 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:19:06.656253 (+ 54us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 23:19:06.656505 (+ 252us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38048: token expired
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 23:19:06.656641 17131 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38048: token expired
W20260924 23:19:06.656699 16871 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.4.254:33901)
I20260924 23:19:06.662842 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.656955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.1.4.254:33901)
0924 23:19:06.656988 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:06.656990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:06.657036 (+ 46us) server_negotiation.cc:407] Connection header received
0924 23:19:06.657127 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:06.657128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:06.657136 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:06.657191 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:06.657665 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.658345 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.659116 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.659308 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.659385 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.660542 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:06.660544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:06.660544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:06.660551 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:06.661725 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:06.662121 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:06.662122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:06.662122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:06.662171 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:06.662339 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:06.662340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:06.662340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:06.662472 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:06.662578 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:06.662710 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 23:19:06.662777 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 23:19:06.664000 16939 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:19:06.665582 16939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:19:06.668025 17062 tablet_service.cc:1467] Processing CreateTablet for tablet 90e5646188454aeaa7c39b477f18f747 (DEFAULT_TABLE table=test-table [id=c9c623cff07d44b4a46d9abcbeba37ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:06.668237 17062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e5646188454aeaa7c39b477f18f747. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:06.669569 17020 tablet_bootstrap.cc:492] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6: Bootstrap starting.
I20260924 23:19:06.669857 17020 tablet_bootstrap.cc:654] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:06.670303 17020 tablet_bootstrap.cc:492] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6: No bootstrap required, opened a new log
I20260924 23:19:06.670351 17020 ts_tablet_manager.cc:1397] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 23:19:06.670467 17020 raft_consensus.cc:348] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } }
I20260924 23:19:06.670538 17020 raft_consensus.cc:374] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:06.670588 17020 raft_consensus.cc:729] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa3ec975b5d4375b0471353b6ede0c6, State: Initialized, Role: FOLLOWER
I20260924 23:19:06.670639 17020 consensus_queue.cc:260] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } }
I20260924 23:19:06.670737 17020 ts_tablet_manager.cc:1428] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:06.671406 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.667464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:39101 (local address 127.0.0.1:46858)
0924 23:19:06.667593 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:06.667601 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:06.667637 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:06.668055 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:06.668056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:06.668057 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:06.668085 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:06.668086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.668980 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.668981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:06.670231 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:06.670233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.671114 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.671115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:06.671180 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.671185 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:19:06.671249 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 23:19:06.671293 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":55,"thread_start_us":84,"threads_started":1}
I20260924 23:19:06.671412 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.667661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.1.4.195:39101)
0924 23:19:06.667843 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:06.667847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:06.667856 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:19:06.667892 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:06.667894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:06.667909 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:06.667981 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:06.668180 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.668902 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.670311 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.671038 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.671108 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.671172 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:06.671265 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 23:19:06.671310 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
I20260924 23:19:06.671489 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.667348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:44769 (local address 127.0.0.1:50888)
0924 23:19:06.667423 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:06.667434 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:06.667478 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:06.667847 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:06.667848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:06.667850 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:06.667893 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:06.667894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.669018 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.669018 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:06.670150 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:06.670152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.671293 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.671294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:06.671350 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.671353 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:19:06.671401 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 23:19:06.671437 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":30}
I20260924 23:19:06.671867 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.667448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50888 (local address 127.1.4.194:44769)
0924 23:19:06.667636 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:06.667641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:06.667652 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:19:06.667693 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:06.667695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:06.667711 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:06.667840 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:06.667955 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.668949 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.670218 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.671200 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.671668 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.671725 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:06.671752 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 23:19:06.671791 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":104,"threads_started":1}
I20260924 23:19:06.672138 17332 tablet_service.cc:1467] Processing CreateTablet for tablet 90e5646188454aeaa7c39b477f18f747 (DEFAULT_TABLE table=test-table [id=c9c623cff07d44b4a46d9abcbeba37ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:06.672472 17332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e5646188454aeaa7c39b477f18f747. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:06.672521 17200 tablet_service.cc:1467] Processing CreateTablet for tablet 90e5646188454aeaa7c39b477f18f747 (DEFAULT_TABLE table=test-table [id=c9c623cff07d44b4a46d9abcbeba37ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:06.672808 17200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e5646188454aeaa7c39b477f18f747. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:06.675889 17409 tablet_bootstrap.cc:492] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab: Bootstrap starting.
I20260924 23:19:06.676482 17409 tablet_bootstrap.cc:654] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:06.676957 17409 log.cc:824] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:06.677240 17408 tablet_bootstrap.cc:492] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209: Bootstrap starting.
I20260924 23:19:06.677614 17409 tablet_bootstrap.cc:492] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab: No bootstrap required, opened a new log
I20260924 23:19:06.677680 17409 ts_tablet_manager.cc:1397] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:19:06.677848 17408 tablet_bootstrap.cc:654] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:06.678275 17408 log.cc:824] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:06.678853 17408 tablet_bootstrap.cc:492] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209: No bootstrap required, opened a new log
I20260924 23:19:06.678908 17408 ts_tablet_manager.cc:1397] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:19:06.679106 17409 raft_consensus.cc:348] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } }
I20260924 23:19:06.679236 17409 raft_consensus.cc:374] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:06.679256 17409 raft_consensus.cc:729] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6c3a74839d3424bbba05b9338eedfab, State: Initialized, Role: FOLLOWER
I20260924 23:19:06.679387 17409 consensus_queue.cc:260] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } }
I20260924 23:19:06.679636 17409 ts_tablet_manager.cc:1428] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:19:06.679670 17266 heartbeater.cc:503] Master 127.1.4.254:33901 was elected leader, sending a full tablet report...
I20260924 23:19:06.680626 17408 raft_consensus.cc:348] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } }
I20260924 23:19:06.680755 17408 raft_consensus.cc:374] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:06.680775 17408 raft_consensus.cc:729] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46fc4161b7144764a9fa0c9e55a9b209, State: Initialized, Role: FOLLOWER
I20260924 23:19:06.680874 17408 consensus_queue.cc:260] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } }
I20260924 23:19:06.681114 17408 ts_tablet_manager.cc:1428] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:19:06.681185 17398 heartbeater.cc:503] Master 127.1.4.254:33901 was elected leader, sending a full tablet report...
I20260924 23:19:06.751473 17413 raft_consensus.cc:482] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:19:06.751612 17413 raft_consensus.cc:504] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } }
I20260924 23:19:06.751962 17413 leader_election.cc:288] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769), 0fa3ec975b5d4375b0471353b6ede0c6 (127.1.4.193:45855)
I20260924 23:19:06.754626 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.752254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:45739 (local address 127.1.4.194:44769)
0924 23:19:06.752307 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:06.752309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:06.752317 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:19:06.752345 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:06.752346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:06.752357 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:06.752405 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:06.752548 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.753013 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.753930 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.754309 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.754369 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.754380 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:06.754524 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 23:19:06.754562 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 23:19:06.754626 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.752141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:44769 (local address 127.1.4.195:45739)
0924 23:19:06.752187 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:06.752201 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:06.752262 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:06.752415 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:06.752416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:06.752417 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:06.752485 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:06.752487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.753089 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.753091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:06.753836 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:06.753838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.754378 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.754379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:06.754460 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.754463 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:19:06.754508 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 23:19:06.754549 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 23:19:06.754750 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.752326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:51603 (local address 127.1.4.193:45855)
0924 23:19:06.752375 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:06.752377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:06.752510 (+ 133us) server_negotiation.cc:407] Connection header received
0924 23:19:06.752555 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:06.752556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:06.752568 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:06.752615 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:06.752764 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.753224 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.754090 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.754450 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.754506 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.754516 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:06.754645 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 23:19:06.754693 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:19:06.754750 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.752264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:45855 (local address 127.1.4.195:51603)
0924 23:19:06.752434 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:06.752445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:06.752503 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:06.752621 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:06.752621 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:06.752623 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:06.752690 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:06.752693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.753291 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.753292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:06.754010 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:06.754011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.754516 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.754517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:06.754585 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.754588 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:19:06.754631 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 23:19:06.754669 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":111,"threads_started":1}
I20260924 23:19:06.755016 17220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e5646188454aeaa7c39b477f18f747" candidate_uuid: "46fc4161b7144764a9fa0c9e55a9b209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c3a74839d3424bbba05b9338eedfab" is_pre_election: true
I20260924 23:19:06.755177 17220 raft_consensus.cc:2454] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46fc4161b7144764a9fa0c9e55a9b209 in term 0.
I20260924 23:19:06.755219 17082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e5646188454aeaa7c39b477f18f747" candidate_uuid: "46fc4161b7144764a9fa0c9e55a9b209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" is_pre_election: true
I20260924 23:19:06.755419 17082 raft_consensus.cc:2454] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46fc4161b7144764a9fa0c9e55a9b209 in term 0.
I20260924 23:19:06.755441 17286 leader_election.cc:302] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fc4161b7144764a9fa0c9e55a9b209, d6c3a74839d3424bbba05b9338eedfab; no voters:
I20260924 23:19:06.755564 17413 raft_consensus.cc:2781] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:19:06.755625 17413 raft_consensus.cc:482] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:19:06.755648 17413 raft_consensus.cc:3037] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:06.756317 17413 raft_consensus.cc:504] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } }
I20260924 23:19:06.756428 17413 leader_election.cc:288] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [CANDIDATE]: Term 1 election: Requested vote from peers d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769), 0fa3ec975b5d4375b0471353b6ede0c6 (127.1.4.193:45855)
I20260924 23:19:06.756590 17220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e5646188454aeaa7c39b477f18f747" candidate_uuid: "46fc4161b7144764a9fa0c9e55a9b209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6c3a74839d3424bbba05b9338eedfab"
I20260924 23:19:06.756668 17220 raft_consensus.cc:3037] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:06.757328 17220 raft_consensus.cc:2454] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fc4161b7144764a9fa0c9e55a9b209 in term 1.
I20260924 23:19:06.757417 17082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e5646188454aeaa7c39b477f18f747" candidate_uuid: "46fc4161b7144764a9fa0c9e55a9b209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fa3ec975b5d4375b0471353b6ede0c6"
I20260924 23:19:06.757483 17286 leader_election.cc:302] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46fc4161b7144764a9fa0c9e55a9b209, d6c3a74839d3424bbba05b9338eedfab; no voters:
I20260924 23:19:06.757498 17082 raft_consensus.cc:3037] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:06.757562 17413 raft_consensus.cc:2781] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:06.757725 17413 raft_consensus.cc:686] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [term 1 LEADER]: Becoming Leader. State: Replica: 46fc4161b7144764a9fa0c9e55a9b209, State: Running, Role: LEADER
I20260924 23:19:06.757875 17413 consensus_queue.cc:237] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } }
I20260924 23:19:06.758267 17082 raft_consensus.cc:2454] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46fc4161b7144764a9fa0c9e55a9b209 in term 1.
I20260924 23:19:06.758741 16939 catalog_manager.cc:5495] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46fc4161b7144764a9fa0c9e55a9b209 (127.1.4.195). New cstate: current_term: 1 leader_uuid: "46fc4161b7144764a9fa0c9e55a9b209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } health_report { overall_health: UNKNOWN } } }
W20260924 23:19:06.761812 17267 tablet.cc:2369] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:19:06.764894 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.762580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46868 (local address 127.1.4.195:39101)
0924 23:19:06.762617 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:06.762619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:06.762659 (+ 40us) server_negotiation.cc:407] Connection header received
0924 23:19:06.762820 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:06.762821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:06.762836 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:06.762883 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:19:06.762961 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.763485 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.764235 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:06.764430 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:06.764484 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:06.764649 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:19:06.764739 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:19:06.764796 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:06.764812 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 23:19:06.764828 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:19:06.766590 17220 raft_consensus.cc:1261] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab [term 1 FOLLOWER]: Refusing update from remote peer 46fc4161b7144764a9fa0c9e55a9b209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:19:06.766604 17082 raft_consensus.cc:1261] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Refusing update from remote peer 46fc4161b7144764a9fa0c9e55a9b209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:19:06.766923 17415 consensus_queue.cc:1035] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:19:06.767026 17413 consensus_queue.cc:1035] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:19:06.769192 17417 mvcc.cc:205] Tried to move back new op lower bound from 7333035813953372160 to 7333035813920890880. Current Snapshot: MvccSnapshot[applied={T|T < 7333035813953372160}]
I20260924 23:19:06.770105 17418 mvcc.cc:205] Tried to move back new op lower bound from 7333035813953372160 to 7333035813920890880. Current Snapshot: MvccSnapshot[applied={T|T < 7333035813953372160}]
I20260924 23:19:06.771415 16939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:19:06.771513 16939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:19:06.772220 16939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Sending DeleteTablet for 3 replicas of tablet 90e5646188454aeaa7c39b477f18f747
I20260924 23:19:06.772564 17200 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e5646188454aeaa7c39b477f18f747 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:06 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:50888
I20260924 23:19:06.772590 17332 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e5646188454aeaa7c39b477f18f747 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:06 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:46858
I20260924 23:19:06.772617 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 16978
I20260924 23:19:06.773255 17428 tablet_replica.cc:330] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab: stopping tablet replica
I20260924 23:19:06.773357 17428 raft_consensus.cc:2229] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:19:06.773466 17428 raft_consensus.cc:2258] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:19:06.773626 17427 tablet_replica.cc:330] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209: stopping tablet replica
I20260924 23:19:06.773713 17427 raft_consensus.cc:2229] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:19:06.773825 17427 raft_consensus.cc:2258] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:19:06.774232 17428 ts_tablet_manager.cc:1905] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:19:06.774410 17427 ts_tablet_manager.cc:1905] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:19:06.776021 17428 ts_tablet_manager.cc:1918] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:19:06.776086 17428 log.cc:1196] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/90e5646188454aeaa7c39b477f18f747
I20260924 23:19:06.776244 17427 ts_tablet_manager.cc:1918] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:19:06.776362 17427 log.cc:1196] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/90e5646188454aeaa7c39b477f18f747
I20260924 23:19:06.776365 17428 ts_tablet_manager.cc:1939] T 90e5646188454aeaa7c39b477f18f747 P d6c3a74839d3424bbba05b9338eedfab: Deleting consensus metadata
I20260924 23:19:06.776628 17427 ts_tablet_manager.cc:1939] T 90e5646188454aeaa7c39b477f18f747 P 46fc4161b7144764a9fa0c9e55a9b209: Deleting consensus metadata
I20260924 23:19:06.777027 16925 catalog_manager.cc:4841] TS 46fc4161b7144764a9fa0c9e55a9b209 (127.1.4.195:39101): tablet 90e5646188454aeaa7c39b477f18f747 (table test-table [id=c9c623cff07d44b4a46d9abcbeba37ab]) successfully deleted
I20260924 23:19:06.777114 16925 catalog_manager.cc:4841] TS d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769): tablet 90e5646188454aeaa7c39b477f18f747 (table test-table [id=c9c623cff07d44b4a46d9abcbeba37ab]) successfully deleted
W20260924 23:19:06.780014 16925 connection.cc:535] client connection to 127.1.4.193:45855 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 23:19:06.780128 16925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 23:19:06.780472 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 17132
I20260924 23:19:06.780565 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:06.780448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:45855 (local address 127.0.0.1:39732)
0924 23:19:06.780497 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:06.780533 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:45855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 23:19:06.781198 16925 catalog_manager.cc:4584] TS 0fa3ec975b5d4375b0471353b6ede0c6 (127.1.4.193:45855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90e5646188454aeaa7c39b477f18f747: Network error: Client connection negotiation failed: client connection to 127.1.4.193:45855: connect: Connection refused (error 111)
I20260924 23:19:06.786100 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 17269
I20260924 23:19:06.790144 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 16907
I20260924 23:19:26.794879 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:33901
--webserver_interface=127.1.4.254
--webserver_port=42041
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:33901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:26.876153 17430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:26.876395 17430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:26.876430 17430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:26.877597 17430 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 23:19:26.877636 17430 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 23:19:26.877652 17430 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 23:19:26.877667 17430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:19:26.877682 17430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:26.877699 17430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:26.877720 17430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:19:26.877740 17430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:19:26.879230 17430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:33901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:33901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=42041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:26.879495 17430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:26.879707 17430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:26.884413 17436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:26.884428 17438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:26.884626 17430 server_base.cc:1034] running on GCE node
W20260924 23:19:26.884431 17435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:26.884925 17430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:26.885201 17430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:26.886341 17430 hybrid_clock.cc:648] HybridClock initialized: now 1790291966886327 us; error 28 us; skew 500 ppm
Sep 24 23:19:26 dist-test-slave-s1hw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291966, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:26.893852 17430 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:19:26.894371 17430 webserver.cc:460] Webserver started at http://127.1.4.254:42041/ using document root <none> and password file <none>
I20260924 23:19:26.894579 17430 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:26.894629 17430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:26.896090 17430 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:26.896907 17445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:26.897091 17430 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:26.897176 17430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "67ac7572fd034c6eae1337ee7ae185cf"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "61c410df8ed17178767ab6a3352d484f"
server_key_iv: "59dcb1fd36487ba29d48dc703e029ac4"
server_key_version: "encryptionkey@0"
I20260924 23:19:26.897532 17430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:26.911598 17430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:26.913785 17430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:26.913928 17430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:26.917863 17430 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:33901
I20260924 23:19:26.917901 17497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:33901 every 8 connection(s)
I20260924 23:19:26.918265 17430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:19:26.919790 17498 sys_catalog.cc:263] Verifying existing consensus state
I20260924 23:19:26.919864 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 17430
I20260924 23:19:26.920117 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:45855
--local_ip_for_outbound_sockets=127.1.4.193
--tserver_master_addrs=127.1.4.254:33901
--webserver_port=46709
--webserver_interface=127.1.4.193
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:19:26.920490 17498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Bootstrap starting.
I20260924 23:19:26.930631 17498 log.cc:824] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:26.933693 17498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:19:26.934031 17498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Bootstrap complete.
I20260924 23:19:26.937203 17498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } }
I20260924 23:19:26.937533 17498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67ac7572fd034c6eae1337ee7ae185cf, State: Initialized, Role: FOLLOWER
I20260924 23:19:26.937705 17498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } }
I20260924 23:19:26.937774 17498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:19:26.937819 17498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:19:26.937855 17498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 2 FOLLOWER]: Advancing to term 3
I20260924 23:19:26.939200 17498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } }
I20260924 23:19:26.939316 17498 leader_election.cc:302] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67ac7572fd034c6eae1337ee7ae185cf; no voters:
I20260924 23:19:26.939538 17498 leader_election.cc:288] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 23:19:26.939628 17502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 3 FOLLOWER]: Leader election won for term 3
I20260924 23:19:26.939785 17502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [term 3 LEADER]: Becoming Leader. State: Replica: 67ac7572fd034c6eae1337ee7ae185cf, State: Running, Role: LEADER
I20260924 23:19:26.939832 17498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:19:26.939881 17502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } }
I20260924 23:19:26.940212 17503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "67ac7572fd034c6eae1337ee7ae185cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } } }
I20260924 23:19:26.940285 17503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:26.940517 17504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67ac7572fd034c6eae1337ee7ae185cf. Latest consensus state: current_term: 3 leader_uuid: "67ac7572fd034c6eae1337ee7ae185cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ac7572fd034c6eae1337ee7ae185cf" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33901 } } }
I20260924 23:19:26.940650 17504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:26.940729 17509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:19:26.941540 17509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:19:26.941640 17509 catalog_manager.cc:1259] Loaded cluster ID: f12ad3691aac4f0da76d3feae0e141ea
I20260924 23:19:26.941673 17509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:19:26.942862 17509 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:19:26.943046 17509 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Loaded TSK: 0
I20260924 23:19:26.943094 17509 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 67ac7572fd034c6eae1337ee7ae185cf: Loaded TSK: 1
F20260924 23:19:26.943326 17509 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f26c92519ad google::LogMessage::Fail() at ??:0
@ 0x7f26c925488a google::LogMessage::SendToLog() at ??:0
@ 0x7f26c92514de google::LogMessage::Flush() at ??:0
@ 0x7f26c9252a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f26cb9be104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f26cb9bbd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f26d11ab1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f26d11ab48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f26d11ab594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f26d1171c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f26d117bad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f26cb11f0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f26cb114954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f26cbf786db start_thread at ??:0
@ 0x7f26c7d4f61f clone at ??:0
W20260924 23:19:27.034080 17500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:27.034387 17500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:27.034430 17500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:27.036267 17500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:27.036345 17500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:27.036388 17500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:19:27.038723 17500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:45855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=46709
--tserver_master_addrs=127.1.4.254:33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:27.039018 17500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:27.039301 17500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:27.044934 17539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:27.045058 17500 server_base.cc:1034] running on GCE node
W20260924 23:19:27.044916 17540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:27.045176 17542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:27.045471 17500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:27.045776 17500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:27.046954 17500 hybrid_clock.cc:648] HybridClock initialized: now 1790291967046931 us; error 39 us; skew 500 ppm
Sep 24 23:19:27 dist-test-slave-s1hw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291967, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:27.049858 17500 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:19:27.050273 17500 webserver.cc:460] Webserver started at http://127.1.4.193:46709/ using document root <none> and password file <none>
I20260924 23:19:27.050503 17500 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:27.050552 17500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:27.052038 17500 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 23:19:27.052829 17549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:27.053061 17500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:27.053160 17500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "0fa3ec975b5d4375b0471353b6ede0c6"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "35a84eae747f2e56d6f5034ef1fc5038"
server_key_iv: "6f094ef528364d8e72d8481970159585"
server_key_version: "encryptionkey@0"
I20260924 23:19:27.053519 17500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:27.071132 17500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:27.073357 17500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:27.073498 17500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:27.073724 17500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:27.074162 17556 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 23:19:27.076216 17500 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 23:19:27.076278 17500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 23:19:27.076314 17500 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 23:19:27.077096 17500 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 23:19:27.077129 17500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 23:19:27.077222 17556 tablet_bootstrap.cc:492] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6: Bootstrap starting.
I20260924 23:19:27.083841 17500 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:45855
I20260924 23:19:27.084220 17500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:19:27.087225 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 17500
I20260924 23:19:27.087514 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:44769
--local_ip_for_outbound_sockets=127.1.4.194
--tserver_master_addrs=127.1.4.254:33901
--webserver_port=42849
--webserver_interface=127.1.4.194
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:19:27.087762 17556 log.cc:824] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:27.088093 17663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:45855 every 8 connection(s)
I20260924 23:19:27.090569 17556 tablet_bootstrap.cc:492] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:19:27.090860 17556 tablet_bootstrap.cc:492] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6: Bootstrap complete.
I20260924 23:19:27.091962 17556 ts_tablet_manager.cc:1397] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20260924 23:19:27.093398 17556 raft_consensus.cc:348] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } }
I20260924 23:19:27.093724 17556 raft_consensus.cc:729] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa3ec975b5d4375b0471353b6ede0c6, State: Initialized, Role: FOLLOWER
I20260924 23:19:27.093895 17556 consensus_queue.cc:260] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } }
I20260924 23:19:27.094164 17556 ts_tablet_manager.cc:1428] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260924 23:19:27.152004 17666 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:19:27.084882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:44709)
0924 23:19:27.085049 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:27.085093 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:27.085539 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:27.151772 (+ 66233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
W20260924 23:19:27.152298 17664 heartbeater.cc:644] Failed to heartbeat to 127.1.4.254:33901 (0 consecutive failures): Network error: Failed to ping master at 127.1.4.254:33901: Client connection negotiation failed: client connection to 127.1.4.254:33901: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 23:19:27.152657 17666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:27.152528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:38181)
0924 23:19:27.152580 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:27.152634 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 23:19:27.153096 17666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:27.152991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:48233)
0924 23:19:27.153033 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:27.153060 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 23:19:27.153203 17664 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 23:19:27.171092 17668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:27.171319 17668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:27.171343 17668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:27.172534 17668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:27.172581 17668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:27.172607 17668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:19:27.174086 17668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:44769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=42849
--tserver_master_addrs=127.1.4.254:33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:27.174309 17668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:27.174503 17668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:27.179499 17675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:27.179557 17668 server_base.cc:1034] running on GCE node
W20260924 23:19:27.179586 17674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:27.179500 17677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:27.179853 17668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:27.180064 17668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:27.181233 17668 hybrid_clock.cc:648] HybridClock initialized: now 1790291967181232 us; error 37 us; skew 500 ppm
Sep 24 23:19:27 dist-test-slave-s1hw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291967, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:27.183686 17668 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:19:27.184087 17668 webserver.cc:460] Webserver started at http://127.1.4.194:42849/ using document root <none> and password file <none>
I20260924 23:19:27.184270 17668 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:27.184312 17668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:27.185561 17668 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:27.186108 17684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:27.186287 17668 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:27.186374 17668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "d6c3a74839d3424bbba05b9338eedfab"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "ffab8a0e44cfdbb89aa08ac0f48322d0"
server_key_iv: "0381004402c971fa1cdf77cebbd43519"
server_key_version: "encryptionkey@0"
I20260924 23:19:27.186738 17668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:27.199134 17668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:27.201272 17668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:27.201385 17668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:27.201622 17668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:27.201915 17668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:27.201949 17668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:27.201972 17668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:27.201984 17668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:27.207950 17668 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:44769
I20260924 23:19:27.208015 17797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:44769 every 8 connection(s)
I20260924 23:19:27.208343 17668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:19:27.209367 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:27.208982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:48029)
0924 23:19:27.209126 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:27.209225 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
W20260924 23:19:27.209722 17798 heartbeater.cc:644] Failed to heartbeat to 127.1.4.254:33901 (0 consecutive failures): Network error: Failed to ping master at 127.1.4.254:33901: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
I20260924 23:19:27.209968 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:27.209892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:39321)
0924 23:19:27.209927 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:27.209953 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 23:19:27.210165 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:27.210108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:47703)
0924 23:19:27.210136 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:27.210153 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 23:19:27.210269 17798 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 23:19:27.213891 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 17668
I20260924 23:19:27.214148 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:39101
--local_ip_for_outbound_sockets=127.1.4.195
--tserver_master_addrs=127.1.4.254:33901
--webserver_port=43747
--webserver_interface=127.1.4.195
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:27.294533 17801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:27.294759 17801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:27.294783 17801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:27.296129 17801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:27.296193 17801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:27.296221 17801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:19:27.297812 17801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:39101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=43747
--tserver_master_addrs=127.1.4.254:33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:27.298054 17801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:27.298292 17801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:27.303629 17809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:27.303644 17807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:27.303628 17806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:27.303802 17801 server_base.cc:1034] running on GCE node
I20260924 23:19:27.304111 17801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:27.304329 17801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:27.305477 17801 hybrid_clock.cc:648] HybridClock initialized: now 1790291967305461 us; error 31 us; skew 500 ppm
Sep 24 23:19:27 dist-test-slave-s1hw krb5kdc[16361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291967, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:27.308028 17801 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:19:27.308466 17801 webserver.cc:460] Webserver started at http://127.1.4.195:43747/ using document root <none> and password file <none>
I20260924 23:19:27.308650 17801 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:27.308696 17801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:27.309986 17801 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:27.310541 17816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:27.310736 17801 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:27.310812 17801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "46fc4161b7144764a9fa0c9e55a9b209"
format_stamp: "Formatted at 2026-09-24 23:18:45 on dist-test-slave-s1hw"
server_key: "c2b943f737541b166436f324181e82b5"
server_key_iv: "6edbd7631ab85d93f273e40f11a0652b"
server_key_version: "encryptionkey@0"
I20260924 23:19:27.311139 17801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:27.340559 17801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:27.342464 17801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:27.342579 17801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:27.342780 17801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:27.343080 17801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:27.343113 17801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:27.343137 17801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:27.343158 17801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:27.348706 17801 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:39101
I20260924 23:19:27.348752 17929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:39101 every 8 connection(s)
I20260924 23:19:27.349063 17801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:19:27.350039 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:27.349671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:49529)
0924 23:19:27.349828 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:27.349897 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
W20260924 23:19:27.350462 17930 heartbeater.cc:644] Failed to heartbeat to 127.1.4.254:33901 (0 consecutive failures): Network error: Failed to ping master at 127.1.4.254:33901: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
I20260924 23:19:27.350759 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:27.350664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:48305)
0924 23:19:27.350712 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:27.350740 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 23:19:27.351054 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:27.350938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:54455)
0924 23:19:27.351003 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:27.351034 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 23:19:27.351188 17930 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 23:19:27.359282 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 17801
I20260924 23:19:28.153926 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:28.153653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:43925)
0924 23:19:28.153815 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:28.153863 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
I20260924 23:19:28.210891 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:28.210624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:34079)
0924 23:19:28.210794 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:28.210836 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":61,"threads_started":1}
I20260924 23:19:28.351840 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:28.351546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:39599)
0924 23:19:28.351742 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:28.351784 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260924 23:19:28.514559 17937 raft_consensus.cc:482] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:19:28.514676 17937 raft_consensus.cc:504] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } }
I20260924 23:19:28.514937 17937 leader_election.cc:288] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769), 46fc4161b7144764a9fa0c9e55a9b209 (127.1.4.195:39101)
Sep 24 23:19:28 dist-test-slave-s1hw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291967, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.194@KRBTEST.COM
Sep 24 23:19:28 dist-test-slave-s1hw krb5kdc[16361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291967, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.195@KRBTEST.COM
I20260924 23:19:28.524737 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:28.515317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:34671 (local address 127.1.4.195:39101)
0924 23:19:28.515451 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:28.515462 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:28.515586 (+ 124us) server_negotiation.cc:407] Connection header received
0924 23:19:28.516071 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:28.516075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:28.516134 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:28.516284 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:28.516994 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:28.517846 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:28.518797 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:28.519195 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:28.519289 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:28.521725 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:28.521728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:28.521729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:28.521738 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:28.523380 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:28.523668 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:28.523671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:28.523671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:28.523759 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:28.523931 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:28.523931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:28.523932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:28.524164 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:28.524232 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:28.524505 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 23:19:28.524629 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":60,"threads_started":1}
I20260924 23:19:28.524744 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:28.515242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:43067 (local address 127.1.4.194:44769)
0924 23:19:28.515407 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:28.515420 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:28.515454 (+ 34us) server_negotiation.cc:407] Connection header received
0924 23:19:28.516071 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:28.516075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:28.516135 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:28.516284 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:28.516990 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:28.517854 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:28.518801 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:28.519192 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:28.519295 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:28.521395 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:28.521398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:28.521399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:28.521409 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:28.523158 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:28.523608 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:28.523612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:28.523612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:28.523715 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:28.523928 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:28.523929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:28.523929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:28.524187 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:28.524257 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:28.524503 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 23:19:28.524622 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260924 23:19:28.524878 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:28.515228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:39101 (local address 127.1.4.193:34671)
0924 23:19:28.515525 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:28.515532 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:28.515577 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:28.516407 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:28.516408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:28.516667 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:28.516881 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:28.516887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:28.517937 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:28.517938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:28.518724 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:28.518726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:28.519277 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:28.519278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:28.519406 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:28.519907 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:19:28.519915 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:28.521638 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:28.523439 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:28.523440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:28.523440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:28.523600 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:28.523823 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:28.523824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:28.523824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:28.523881 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:28.524225 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:28.524232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:28.524356 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 23:19:28.524445 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 23:19:28.524769 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":241,"thread_start_us":59,"threads_started":1}
I20260924 23:19:28.524798 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:28.515108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:44769 (local address 127.1.4.193:43067)
0924 23:19:28.515181 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:28.515195 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:28.515270 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:28.516341 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:28.516344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:28.516669 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:28.516881 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:28.516887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:28.517962 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:28.517963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:28.518722 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:28.518724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:28.519274 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:28.519275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:28.519407 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:28.519954 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:19:28.519958 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:28.521296 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:28.523245 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:28.523251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:28.523253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:28.523530 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:28.523775 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:28.523776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:28.523776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:28.523872 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:28.524252 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:28.524253 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:28.524357 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 23:19:28.524444 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 23:19:28.524706 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":45}
W20260924 23:19:28.525758 17551 leader_election.cc:341] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769): Not found: Tablet not found: 90e5646188454aeaa7c39b477f18f747
W20260924 23:19:28.526051 17551 leader_election.cc:341] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46fc4161b7144764a9fa0c9e55a9b209 (127.1.4.195:39101): Not found: Tablet not found: 90e5646188454aeaa7c39b477f18f747
I20260924 23:19:28.526114 17551 leader_election.cc:302] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fa3ec975b5d4375b0471353b6ede0c6; no voters: 46fc4161b7144764a9fa0c9e55a9b209, d6c3a74839d3424bbba05b9338eedfab
I20260924 23:19:28.526255 17937 raft_consensus.cc:2726] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:19:29.154680 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:29.154441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:48975)
0924 23:19:29.154586 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:29.154633 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260924 23:19:29.211515 17942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:29.211312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:58527)
0924 23:19:29.211442 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:29.211476 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260924 23:19:29.352538 17943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:29.352307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:57821)
0924 23:19:29.352456 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:29.352495 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260924 23:19:30.155579 17944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:30.155284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:34279)
0924 23:19:30.155476 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:30.155529 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":100,"threads_started":1}
I20260924 23:19:30.212167 17945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:30.211930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:54881)
0924 23:19:30.212073 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:30.212121 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":69,"threads_started":1}
I20260924 23:19:30.353242 17946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:30.353023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:49479)
0924 23:19:30.353155 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:30.353199 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260924 23:19:30.685601 17947 raft_consensus.cc:482] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:19:30.685706 17947 raft_consensus.cc:504] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } }
I20260924 23:19:30.685830 17947 leader_election.cc:288] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769), 46fc4161b7144764a9fa0c9e55a9b209 (127.1.4.195:39101)
W20260924 23:19:30.686371 17551 leader_election.cc:341] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769): Not found: Tablet not found: 90e5646188454aeaa7c39b477f18f747
W20260924 23:19:30.686451 17551 leader_election.cc:341] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46fc4161b7144764a9fa0c9e55a9b209 (127.1.4.195:39101): Not found: Tablet not found: 90e5646188454aeaa7c39b477f18f747
I20260924 23:19:30.686475 17551 leader_election.cc:302] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fa3ec975b5d4375b0471353b6ede0c6; no voters: 46fc4161b7144764a9fa0c9e55a9b209, d6c3a74839d3424bbba05b9338eedfab
I20260924 23:19:30.686555 17947 raft_consensus.cc:2726] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:19:31.156319 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:31.156079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:39079)
0924 23:19:31.156227 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:31.156272 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260924 23:19:31.212879 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:31.212656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:53009)
0924 23:19:31.212786 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:31.212832 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260924 23:19:31.353926 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:31.353706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:37787)
0924 23:19:31.353840 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:31.353882 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260924 23:19:32.157078 17951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:32.156861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:44993)
0924 23:19:32.156976 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:32.157026 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260924 23:19:32.213578 17952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:32.213359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:54663)
0924 23:19:32.213485 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:32.213532 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":56,"threads_started":1}
I20260924 23:19:32.354617 17953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:32.354393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:59011)
0924 23:19:32.354528 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:32.354574 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
I20260924 23:19:33.157859 17954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:33.157604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:33875)
0924 23:19:33.157755 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:33.157800 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260924 23:19:33.214262 17955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:33.214014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:36665)
0924 23:19:33.214151 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:33.214217 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260924 23:19:33.355348 17956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:33.355124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:43491)
0924 23:19:33.355258 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:33.355301 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260924 23:19:33.997890 17957 raft_consensus.cc:482] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:19:33.997988 17957 raft_consensus.cc:504] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } }
I20260924 23:19:33.998118 17957 leader_election.cc:288] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769), 46fc4161b7144764a9fa0c9e55a9b209 (127.1.4.195:39101)
W20260924 23:19:33.998551 17551 leader_election.cc:341] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46fc4161b7144764a9fa0c9e55a9b209 (127.1.4.195:39101): Not found: Tablet not found: 90e5646188454aeaa7c39b477f18f747
W20260924 23:19:33.998612 17551 leader_election.cc:341] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769): Not found: Tablet not found: 90e5646188454aeaa7c39b477f18f747
I20260924 23:19:33.998625 17551 leader_election.cc:302] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fa3ec975b5d4375b0471353b6ede0c6; no voters: 46fc4161b7144764a9fa0c9e55a9b209, d6c3a74839d3424bbba05b9338eedfab
I20260924 23:19:33.998756 17957 raft_consensus.cc:2726] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:19:34.158568 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:34.158333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:44319)
0924 23:19:34.158472 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:34.158521 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260924 23:19:34.214869 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:34.214661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:46913)
0924 23:19:34.214788 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:34.214826 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260924 23:19:34.356073 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:34.355846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:34865)
0924 23:19:34.355964 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:34.356007 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260924 23:19:35.159327 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:35.159091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:57997)
0924 23:19:35.159231 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:35.159279 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260924 23:19:35.215605 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:35.215392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:51919)
0924 23:19:35.215519 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:35.215558 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260924 23:19:35.356738 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:35.356524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:34675)
0924 23:19:35.356646 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:35.356691 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260924 23:19:35.780033 17964 raft_consensus.cc:482] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:19:35.780135 17964 raft_consensus.cc:504] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } }
I20260924 23:19:35.780283 17964 leader_election.cc:288] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769), 46fc4161b7144764a9fa0c9e55a9b209 (127.1.4.195:39101)
W20260924 23:19:35.780691 17551 leader_election.cc:341] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769): Not found: Tablet not found: 90e5646188454aeaa7c39b477f18f747
W20260924 23:19:35.780771 17551 leader_election.cc:341] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46fc4161b7144764a9fa0c9e55a9b209 (127.1.4.195:39101): Not found: Tablet not found: 90e5646188454aeaa7c39b477f18f747
I20260924 23:19:35.780794 17551 leader_election.cc:302] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fa3ec975b5d4375b0471353b6ede0c6; no voters: 46fc4161b7144764a9fa0c9e55a9b209, d6c3a74839d3424bbba05b9338eedfab
I20260924 23:19:35.780864 17964 raft_consensus.cc:2726] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:19:36.160120 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:36.159876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:44695)
0924 23:19:36.160016 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:36.160064 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260924 23:19:36.216266 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:36.216036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:58645)
0924 23:19:36.216172 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:36.216212 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260924 23:19:36.357409 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:36.357190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:55545)
0924 23:19:36.357320 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:36.357362 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260924 23:19:37.160908 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:37.160679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:53613)
0924 23:19:37.160820 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:37.160863 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260924 23:19:37.216993 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:37.216753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:50247)
0924 23:19:37.216899 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:37.216946 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":59,"threads_started":1}
I20260924 23:19:37.358134 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:37.357912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:51629)
0924 23:19:37.358046 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:37.358080 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260924 23:19:38.161667 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:38.161434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:48147)
0924 23:19:38.161588 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:38.161621 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20260924 23:19:38.217788 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:38.217528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:33799)
0924 23:19:38.217694 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:38.217741 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260924 23:19:38.358901 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:38.358652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:46175)
0924 23:19:38.358806 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:38.358854 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260924 23:19:39.162429 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:39.162201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:54983)
0924 23:19:39.162334 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:39.162382 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260924 23:19:39.218467 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:39.218238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:57587)
0924 23:19:39.218379 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:39.218420 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260924 23:19:39.359601 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:39.359386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:48785)
0924 23:19:39.359511 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:39.359553 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260924 23:19:40.163170 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:40.162931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:42737)
0924 23:19:40.163074 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:40.163121 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260924 23:19:40.219131 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:40.218930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:52331)
0924 23:19:40.219043 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:40.219082 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
I20260924 23:19:40.360379 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:40.360126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:46341)
0924 23:19:40.360272 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:40.360328 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260924 23:19:41.163967 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:41.163728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:40797)
0924 23:19:41.163871 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:41.163917 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260924 23:19:41.219817 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:41.219586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:47467)
0924 23:19:41.219727 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:41.219770 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260924 23:19:41.361107 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:41.360894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:35695)
0924 23:19:41.361020 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:41.361059 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260924 23:19:41.511354 17983 raft_consensus.cc:482] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:19:41.511487 17983 raft_consensus.cc:504] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6c3a74839d3424bbba05b9338eedfab" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44769 } } peers { permanent_uuid: "46fc4161b7144764a9fa0c9e55a9b209" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39101 } } peers { permanent_uuid: "0fa3ec975b5d4375b0471353b6ede0c6" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 45855 } }
I20260924 23:19:41.511616 17983 leader_election.cc:288] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769), 46fc4161b7144764a9fa0c9e55a9b209 (127.1.4.195:39101)
W20260924 23:19:41.512006 17551 leader_election.cc:341] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d6c3a74839d3424bbba05b9338eedfab (127.1.4.194:44769): Not found: Tablet not found: 90e5646188454aeaa7c39b477f18f747
W20260924 23:19:41.512086 17551 leader_election.cc:341] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46fc4161b7144764a9fa0c9e55a9b209 (127.1.4.195:39101): Not found: Tablet not found: 90e5646188454aeaa7c39b477f18f747
I20260924 23:19:41.512125 17551 leader_election.cc:302] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fa3ec975b5d4375b0471353b6ede0c6; no voters: 46fc4161b7144764a9fa0c9e55a9b209, d6c3a74839d3424bbba05b9338eedfab
I20260924 23:19:41.512205 17983 raft_consensus.cc:2726] T 90e5646188454aeaa7c39b477f18f747 P 0fa3ec975b5d4375b0471353b6ede0c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:19:42.164893 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:42.164651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.193:46435)
0924 23:19:42.164805 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:42.164847 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20260924 23:19:42.220400 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:42.220219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.194:36941)
0924 23:19:42.220323 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:42.220354 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":59,"threads_started":1}
I20260924 23:19:42.362001 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:42.361756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:33901 (local address 127.1.4.195:38875)
0924 23:19:42.361910 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:42.361956 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.254:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260924 23:19:42.365430 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 17500
I20260924 23:19:42.369849 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 17668
I20260924 23:19:42.373924 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 17801
2026-09-24T23:19:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58320 ms)
[----------] 30 tests from SecurityITest (226923 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:19:42 dist-test-slave-s1hw krb5kdc[17989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:19:42 dist-test-slave-s1hw krb5kdc[17989](info): set up 2 sockets
Sep 24 23:19:42 dist-test-slave-s1hw krb5kdc[17989](info): commencing operation
krb5kdc: starting...
W20260924 23:19:43.330843 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:19:43 dist-test-slave-s1hw krb5kdc[17989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:19:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:19:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:19:43.448035 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:44863
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:44863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:43.527256 18002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:43.527487 18002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:43.527510 18002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:43.528693 18002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:19:43.528749 18002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:43.528764 18002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:43.528774 18002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:19:43.528787 18002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:19:43.530267 18002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:44863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:44863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:43.530525 18002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:43.530743 18002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:43.536042 18007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:43.536134 18008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:43.536176 18002 server_base.cc:1034] running on GCE node
W20260924 23:19:43.536093 18010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:43.536453 18002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:43.536830 18002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:43.537989 18002 hybrid_clock.cc:648] HybridClock initialized: now 1790291983537970 us; error 38 us; skew 500 ppm
Sep 24 23:19:43 dist-test-slave-s1hw krb5kdc[17989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291983, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:43.540788 18002 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:19:43.541239 18002 webserver.cc:460] Webserver started at http://127.1.4.254:32793/ using document root <none> and password file <none>
I20260924 23:19:43.541441 18002 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:43.541493 18002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:43.541638 18002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:43.542779 18002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "5367d19c7c6a409daa58c453afd7bcfc"
format_stamp: "Formatted at 2026-09-24 23:19:43 on dist-test-slave-s1hw"
server_key: "fbd94f4226ad4d2a89a6ced163bbe1b6"
server_key_iv: "dafaf701ddc2b6209ca6b4dbb46f5f31"
server_key_version: "encryptionkey@0"
I20260924 23:19:43.543123 18002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "5367d19c7c6a409daa58c453afd7bcfc"
format_stamp: "Formatted at 2026-09-24 23:19:43 on dist-test-slave-s1hw"
server_key: "fbd94f4226ad4d2a89a6ced163bbe1b6"
server_key_iv: "dafaf701ddc2b6209ca6b4dbb46f5f31"
server_key_version: "encryptionkey@0"
I20260924 23:19:43.544514 18002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:19:43.545274 18017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:43.545423 18002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:19:43.545514 18002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "5367d19c7c6a409daa58c453afd7bcfc"
format_stamp: "Formatted at 2026-09-24 23:19:43 on dist-test-slave-s1hw"
server_key: "fbd94f4226ad4d2a89a6ced163bbe1b6"
server_key_iv: "dafaf701ddc2b6209ca6b4dbb46f5f31"
server_key_version: "encryptionkey@0"
I20260924 23:19:43.545576 18002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:43.554528 18002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:43.556644 18002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:43.556785 18002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:43.560369 18002 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:44863
I20260924 23:19:43.560406 18069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:44863 every 8 connection(s)
I20260924 23:19:43.560802 18002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:19:43.561616 18070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:43.562598 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 18002
I20260924 23:19:43.562691 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:19:43.562867 1043 external_mini_cluster.cc:1428] Setting key d1f365680c876700a38ce4fb4991cb9c
I20260924 23:19:43.564107 18070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc: Bootstrap starting.
I20260924 23:19:43.564847 18070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:43.565122 18070 log.cc:824] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:43.565738 18070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc: No bootstrap required, opened a new log
I20260924 23:19:43.567507 18070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5367d19c7c6a409daa58c453afd7bcfc" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 44863 } }
I20260924 23:19:43.567641 18070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:43.567684 18070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5367d19c7c6a409daa58c453afd7bcfc, State: Initialized, Role: FOLLOWER
I20260924 23:19:43.567816 18070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5367d19c7c6a409daa58c453afd7bcfc" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 44863 } }
I20260924 23:19:43.567879 18070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:19:43.567900 18070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:19:43.567932 18070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:43.568504 18070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5367d19c7c6a409daa58c453afd7bcfc" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 44863 } }
I20260924 23:19:43.568616 18070 leader_election.cc:302] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5367d19c7c6a409daa58c453afd7bcfc; no voters:
I20260924 23:19:43.568764 18070 leader_election.cc:288] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:19:43.568813 18075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:43.568975 18075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [term 1 LEADER]: Becoming Leader. State: Replica: 5367d19c7c6a409daa58c453afd7bcfc, State: Running, Role: LEADER
I20260924 23:19:43.569060 18070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 23:19:43 dist-test-slave-s1hw krb5kdc[17989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:19:43.569090 18075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5367d19c7c6a409daa58c453afd7bcfc" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 44863 } }
I20260924 23:19:43.569413 18076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5367d19c7c6a409daa58c453afd7bcfc. Latest consensus state: current_term: 1 leader_uuid: "5367d19c7c6a409daa58c453afd7bcfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5367d19c7c6a409daa58c453afd7bcfc" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 44863 } } }
I20260924 23:19:43.569532 18076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:43.569746 18083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:19:43.570220 18083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:19:43.570163 18077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5367d19c7c6a409daa58c453afd7bcfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5367d19c7c6a409daa58c453afd7bcfc" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 44863 } } }
I20260924 23:19:43.571166 18077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:43.572003 18083 catalog_manager.cc:1347] Generated new cluster ID: e511f6318f034b06b1fd90de6cf6f620
I20260924 23:19:43.572045 18083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:19:43.573563 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:43.563350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.1.4.254:44863)
0924 23:19:43.563516 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:43.563523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:43.563548 (+ 25us) server_negotiation.cc:407] Connection header received
0924 23:19:43.563962 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:43.563969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:43.564020 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:43.564131 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:43.564875 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:43.565828 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:43.566559 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:43.567014 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:43.567115 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:43.569684 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:43.569688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:43.569689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:43.569701 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:43.571652 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:43.572050 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:43.572053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:43.572054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:43.572162 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:43.572355 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:43.572356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:43.572356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:43.572642 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:43.572793 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:43.573223 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 23:19:43.573362 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260924 23:19:43.583879 18083 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:19:43.584617 18083 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:19:43.591226 18083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc: Generated new TSK 0
I20260924 23:19:43.591467 18083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:19:43.618028 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:44863
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:43.709211 18096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:43.709424 18096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:43.709456 18096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:43.710644 18096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:43.710687 18096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:43.710726 18096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:19:43.712571 18096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:44863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:43.712831 18096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:43.713044 18096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:43.717978 18104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:43.718000 18096 server_base.cc:1034] running on GCE node
W20260924 23:19:43.717964 18102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:43.717969 18101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:43.718371 18096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:43.718595 18096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:43.719720 18096 hybrid_clock.cc:648] HybridClock initialized: now 1790291983719696 us; error 42 us; skew 500 ppm
Sep 24 23:19:43 dist-test-slave-s1hw krb5kdc[17989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291983, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:43.722122 18096 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:19:43.722517 18096 webserver.cc:460] Webserver started at http://127.1.4.193:36927/ using document root <none> and password file <none>
I20260924 23:19:43.722725 18096 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:43.722779 18096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:43.722903 18096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:43.723971 18096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "e4ed3778cddd4810939b05452c64f23d"
format_stamp: "Formatted at 2026-09-24 23:19:43 on dist-test-slave-s1hw"
server_key: "075ee574f35883f87ee9381c8c408b8f"
server_key_iv: "17266d95e4163c1f168ef76c6fcbf765"
server_key_version: "encryptionkey@0"
I20260924 23:19:43.724290 18096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "e4ed3778cddd4810939b05452c64f23d"
format_stamp: "Formatted at 2026-09-24 23:19:43 on dist-test-slave-s1hw"
server_key: "075ee574f35883f87ee9381c8c408b8f"
server_key_iv: "17266d95e4163c1f168ef76c6fcbf765"
server_key_version: "encryptionkey@0"
I20260924 23:19:43.725505 18096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:19:43.726145 18111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:43.726301 18096 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:43.726382 18096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "e4ed3778cddd4810939b05452c64f23d"
format_stamp: "Formatted at 2026-09-24 23:19:43 on dist-test-slave-s1hw"
server_key: "075ee574f35883f87ee9381c8c408b8f"
server_key_iv: "17266d95e4163c1f168ef76c6fcbf765"
server_key_version: "encryptionkey@0"
I20260924 23:19:43.726440 18096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:43.746209 18096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:43.748237 18096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:43.748377 18096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:43.748591 18096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:43.748910 18096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:43.748945 18096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:43.748975 18096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:43.748992 18096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:43.754592 18096 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:39315
I20260924 23:19:43.754649 18224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:39315 every 8 connection(s)
I20260924 23:19:43.754971 18096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:19:43 dist-test-slave-s1hw krb5kdc[17989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291983, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:19:43.763561 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 18096
I20260924 23:19:43.763695 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:19:43.763887 1043 external_mini_cluster.cc:1428] Setting key 2d74cf5ed972a9d254c31236a66aa1a5
I20260924 23:19:43.764212 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:43.755668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:37393 (local address 127.1.4.254:44863)
0924 23:19:43.755756 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:43.755758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:43.755960 (+ 202us) server_negotiation.cc:407] Connection header received
0924 23:19:43.756483 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:43.756485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:43.756499 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:43.756557 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:43.757530 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:43.758049 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:43.759069 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:43.759301 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:43.759393 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:43.761510 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:43.761511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:43.761512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:43.761518 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:43.762752 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:43.763148 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:43.763149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:43.763150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:43.763210 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:43.763461 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:43.763462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:43.763463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:43.763620 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:43.763701 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:43.764012 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 23:19:43.764126 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 23:19:43.764394 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:43.755662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:44863 (local address 127.1.4.193:37393)
0924 23:19:43.755838 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:43.755876 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:43.756350 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:43.756632 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:43.756635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:43.757215 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:43.757424 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:43.757429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:43.758144 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:43.758145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:43.758970 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:43.758972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:43.759403 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:43.759405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:43.759520 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:43.759981 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:19:43.759989 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:43.761410 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:43.762825 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:43.762833 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:43.762835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:43.763055 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:43.763271 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:43.763272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:43.763272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:43.763400 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:43.763703 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:43.763710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:43.763887 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 23:19:43.763988 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 23:19:43.764168 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260924 23:19:43.764941 18225 heartbeater.cc:347] Connected to a master server at 127.1.4.254:44863
I20260924 23:19:43.765039 18225 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:43.765232 18225 heartbeater.cc:511] Master 127.1.4.254:44863 requested a full tablet report, sending...
I20260924 23:19:43.765769 18034 ts_manager.cc:194] Registered new tserver with Master: e4ed3778cddd4810939b05452c64f23d (127.1.4.193:39315)
I20260924 23:19:43.766448 18034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:37393
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:19:43.789990 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:44863
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:43.870652 18230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:43.870888 18230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:43.870913 18230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:43.872077 18230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:43.872119 18230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:43.872144 18230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:19:43.873546 18230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:44863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:43.873747 18230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:43.873936 18230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:43.878863 18238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:43.878913 18236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:43.878961 18230 server_base.cc:1034] running on GCE node
W20260924 23:19:43.878924 18235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:43.879308 18230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:43.879555 18230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:43.880749 18230 hybrid_clock.cc:648] HybridClock initialized: now 1790291983880724 us; error 39 us; skew 500 ppm
Sep 24 23:19:43 dist-test-slave-s1hw krb5kdc[17989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291983, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:43.883111 18230 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:19:43.883586 18230 webserver.cc:460] Webserver started at http://127.1.4.194:39773/ using document root <none> and password file <none>
I20260924 23:19:43.883816 18230 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:43.883867 18230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:43.883991 18230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:43.885053 18230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "03e70c3711ba4f07949f36f6d9495597"
format_stamp: "Formatted at 2026-09-24 23:19:43 on dist-test-slave-s1hw"
server_key: "706fded54b1a92292823caa95b84125e"
server_key_iv: "47e2dfeea736415b104988a9793d45ce"
server_key_version: "encryptionkey@0"
I20260924 23:19:43.885397 18230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "03e70c3711ba4f07949f36f6d9495597"
format_stamp: "Formatted at 2026-09-24 23:19:43 on dist-test-slave-s1hw"
server_key: "706fded54b1a92292823caa95b84125e"
server_key_iv: "47e2dfeea736415b104988a9793d45ce"
server_key_version: "encryptionkey@0"
I20260924 23:19:43.886672 18230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:19:43.887439 18245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:43.887593 18230 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:43.887673 18230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "03e70c3711ba4f07949f36f6d9495597"
format_stamp: "Formatted at 2026-09-24 23:19:43 on dist-test-slave-s1hw"
server_key: "706fded54b1a92292823caa95b84125e"
server_key_iv: "47e2dfeea736415b104988a9793d45ce"
server_key_version: "encryptionkey@0"
I20260924 23:19:43.887743 18230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:43.905004 18230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:43.907086 18230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:43.907240 18230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:43.907490 18230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:43.907807 18230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:43.907840 18230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:43.907871 18230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:43.907887 18230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:43.913334 18230 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:45557
I20260924 23:19:43.913393 18358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:45557 every 8 connection(s)
I20260924 23:19:43.913705 18230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:19:43.914613 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 18230
I20260924 23:19:43.914738 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:19:43.914935 1043 external_mini_cluster.cc:1428] Setting key 5a45f4ff6130b8030209e08371ae3874
Sep 24 23:19:43 dist-test-slave-s1hw krb5kdc[17989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291983, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
I20260924 23:19:43.923316 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:43.914392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:32893 (local address 127.1.4.254:44863)
0924 23:19:43.914454 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:43.914456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:43.914677 (+ 221us) server_negotiation.cc:407] Connection header received
0924 23:19:43.915216 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:43.915218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:43.915230 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:43.915281 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:43.916240 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:43.916751 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:43.917819 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:43.918100 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:43.918170 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:43.920563 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:43.920565 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:43.920566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:43.920574 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:43.921994 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:43.922415 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:43.922417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:43.922417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:43.922478 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:43.922686 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:43.922686 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:43.922687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:43.922825 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:43.922892 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:43.923185 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 23:19:43.923249 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 23:19:43.923489 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:43.914355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:44863 (local address 127.1.4.194:32893)
0924 23:19:43.914528 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:43.914569 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:43.915119 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:43.915352 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:43.915356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:43.915945 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:43.916139 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:43.916145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:43.916859 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:43.916860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:43.917729 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:43.917732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:43.918208 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:43.918209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:43.918323 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:43.918906 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:19:43.918917 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:43.920470 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:43.922076 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:43.922082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:43.922084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:43.922336 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:43.922541 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:43.922542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:43.922542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:43.922632 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:43.922889 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:43.922896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:43.923032 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 23:19:43.923144 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 23:19:43.923272 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260924 23:19:43.923985 18359 heartbeater.cc:347] Connected to a master server at 127.1.4.254:44863
I20260924 23:19:43.924079 18359 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:43.924253 18359 heartbeater.cc:511] Master 127.1.4.254:44863 requested a full tablet report, sending...
I20260924 23:19:43.924701 18034 ts_manager.cc:194] Registered new tserver with Master: 03e70c3711ba4f07949f36f6d9495597 (127.1.4.194:45557)
I20260924 23:19:43.925094 18034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:32893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:19:43.941998 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:44863
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:44.021242 18364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:44.021461 18364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:44.021490 18364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:44.022682 18364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:44.022728 18364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:44.022761 18364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:19:44.024266 18364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:44863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:44.024514 18364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:44.024725 18364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:44.029937 18372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:44.030069 18370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:44.030166 18364 server_base.cc:1034] running on GCE node
W20260924 23:19:44.030176 18369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:44.030459 18364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:44.030670 18364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:44.031804 18364 hybrid_clock.cc:648] HybridClock initialized: now 1790291984031786 us; error 33 us; skew 500 ppm
Sep 24 23:19:44 dist-test-slave-s1hw krb5kdc[17989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291984, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:44.034137 18364 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:19:44.034564 18364 webserver.cc:460] Webserver started at http://127.1.4.195:40135/ using document root <none> and password file <none>
I20260924 23:19:44.034781 18364 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:44.034830 18364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:44.034957 18364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:44.036060 18364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "6eaf1133435247348a15fe19ac65a7ad"
format_stamp: "Formatted at 2026-09-24 23:19:44 on dist-test-slave-s1hw"
server_key: "d1274565b440ba7358428cb89a0e46bb"
server_key_iv: "541111860f140027ced20a970429dca5"
server_key_version: "encryptionkey@0"
I20260924 23:19:44.036393 18364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "6eaf1133435247348a15fe19ac65a7ad"
format_stamp: "Formatted at 2026-09-24 23:19:44 on dist-test-slave-s1hw"
server_key: "d1274565b440ba7358428cb89a0e46bb"
server_key_iv: "541111860f140027ced20a970429dca5"
server_key_version: "encryptionkey@0"
I20260924 23:19:44.037647 18364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:19:44.038414 18379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:44.038664 18364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:44.038769 18364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "6eaf1133435247348a15fe19ac65a7ad"
format_stamp: "Formatted at 2026-09-24 23:19:44 on dist-test-slave-s1hw"
server_key: "d1274565b440ba7358428cb89a0e46bb"
server_key_iv: "541111860f140027ced20a970429dca5"
server_key_version: "encryptionkey@0"
I20260924 23:19:44.038830 18364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:44.063886 18364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:44.065836 18364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:44.065976 18364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:44.066195 18364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:44.066502 18364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:44.066535 18364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:44.066566 18364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:44.066589 18364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:44.072414 18364 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:41303
I20260924 23:19:44.072455 18492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:41303 every 8 connection(s)
I20260924 23:19:44.072773 18364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:19:44.077104 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 18364
I20260924 23:19:44.077200 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:19:44.077369 1043 external_mini_cluster.cc:1428] Setting key fb0d6f4f9e6a90597268a692b0246c91
Sep 24 23:19:44 dist-test-slave-s1hw krb5kdc[17989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291984, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:19:44.081642 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.073508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:36723 (local address 127.1.4.254:44863)
0924 23:19:44.073568 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.073570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.073724 (+ 154us) server_negotiation.cc:407] Connection header received
0924 23:19:44.074215 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.074216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.074226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.074262 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:44.075156 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.075692 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.076704 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.076944 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.077018 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.079240 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:44.079241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:44.079242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:44.079248 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:44.080427 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:44.080840 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:44.080841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:44.080842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:44.080895 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:44.081118 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:44.081119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:44.081119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:44.081251 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:44.081306 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.081517 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.081577 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 23:19:44.081897 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.073456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:44863 (local address 127.1.4.195:36723)
0924 23:19:44.073603 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:44.073645 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:44.074118 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:44.074325 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:44.074328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:44.074889 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:44.075086 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.075091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.075797 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.075798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.076594 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.076596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.077023 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.077024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.077127 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.077673 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:19:44.077681 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:44.079157 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:44.080489 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:44.080495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:44.080497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:44.080710 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:44.080954 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:44.080954 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:44.080955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:44.081049 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:44.081302 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:44.081304 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:44.081417 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 23:19:44.081492 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 23:19:44.081611 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260924 23:19:44.082348 18493 heartbeater.cc:347] Connected to a master server at 127.1.4.254:44863
I20260924 23:19:44.082441 18493 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:44.082604 18493 heartbeater.cc:511] Master 127.1.4.254:44863 requested a full tablet report, sending...
I20260924 23:19:44.082970 18034 ts_manager.cc:194] Registered new tserver with Master: 6eaf1133435247348a15fe19ac65a7ad (127.1.4.195:41303)
I20260924 23:19:44.083331 18034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:36723
I20260924 23:19:44.089170 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:19:44.095927 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.090401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.1.4.254:44863)
0924 23:19:44.090451 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.090453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.090585 (+ 132us) server_negotiation.cc:407] Connection header received
0924 23:19:44.090643 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.090643 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.090654 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.090698 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:44.091244 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.091823 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.092505 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.092705 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.092758 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.093817 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:44.093818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:44.093819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:44.093825 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:44.094849 (+ 1024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:44.095144 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:44.095145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:44.095145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:44.095219 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:44.095396 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:44.095397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:44.095397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:44.095549 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:44.095638 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.095761 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.095838 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 23:19:44.097339 18034 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:19:44.098877 18034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:19:44.105314 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.101240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44122 (local address 127.1.4.195:41303)
0924 23:19:44.101943 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.101948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.101958 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:19:44.101999 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.102001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.102016 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.102090 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:44.102202 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.103154 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.104241 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.104941 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.105017 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.105077 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.105187 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.105228 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":68,"threads_started":1}
I20260924 23:19:44.105322 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.100980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:41303 (local address 127.0.0.1:44122)
0924 23:19:44.101229 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:44.101238 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:44.101279 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:44.102104 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:44.102105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:44.102107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:44.102144 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.102146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.103245 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.103246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.104163 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.104165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.105019 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.105020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.105082 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.105094 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 23:19:44.105164 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 23:19:44.105217 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":4,"thread_start_us":77,"threads_started":1}
I20260924 23:19:44.105484 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.101089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.1.4.194:45557)
0924 23:19:44.101635 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.101638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.101645 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:19:44.101678 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.101680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.101728 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.101809 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:44.101930 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.102844 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.104392 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.105063 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.105124 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.105176 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.105364 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.105410 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":67,"threads_started":1}
I20260924 23:19:44.105324 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.100979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:45557 (local address 127.0.0.1:40034)
0924 23:19:44.101373 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:44.101381 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:44.101412 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:44.101807 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:44.101808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:44.101810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:44.101849 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.101850 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.102942 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.102943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.104311 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.104314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.105135 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.105135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.105188 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.105191 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:19:44.105233 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 23:19:44.105268 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":70,"thread_start_us":85,"threads_started":1}
I20260924 23:19:44.106235 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.100979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:39315 (local address 127.0.0.1:43736)
0924 23:19:44.101144 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:44.101185 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:44.101278 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:44.101569 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:44.101571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:44.101572 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:44.101626 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.101627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.102926 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.102927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.104312 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.104314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.106005 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.106006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.106068 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.106072 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:19:44.106130 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 23:19:44.106170 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":34,"thread_start_us":55,"threads_started":1}
I20260924 23:19:44.106261 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.101223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.1.4.193:39315)
0924 23:19:44.101403 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.101408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.101419 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:19:44.101452 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.101454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.101465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.101531 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:44.101701 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.102828 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.105221 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.105923 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.105995 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.106049 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.106143 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.106178 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
I20260924 23:19:44.106328 18427 tablet_service.cc:1467] Processing CreateTablet for tablet 790c12e5d44247faae5e18100b62e760 (DEFAULT_TABLE table=test-table [id=6880b622b54d4e378ab13360d9ef8b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:44.106319 18293 tablet_service.cc:1467] Processing CreateTablet for tablet 790c12e5d44247faae5e18100b62e760 (DEFAULT_TABLE table=test-table [id=6880b622b54d4e378ab13360d9ef8b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:44.106638 18293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 790c12e5d44247faae5e18100b62e760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:44.106649 18427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 790c12e5d44247faae5e18100b62e760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:44.106854 18159 tablet_service.cc:1467] Processing CreateTablet for tablet 790c12e5d44247faae5e18100b62e760 (DEFAULT_TABLE table=test-table [id=6880b622b54d4e378ab13360d9ef8b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:44.107125 18159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 790c12e5d44247faae5e18100b62e760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:44.109057 18510 tablet_bootstrap.cc:492] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597: Bootstrap starting.
I20260924 23:19:44.109609 18510 tablet_bootstrap.cc:654] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:44.109799 18511 tablet_bootstrap.cc:492] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad: Bootstrap starting.
I20260924 23:19:44.110145 18510 log.cc:824] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:44.110358 18512 tablet_bootstrap.cc:492] T 790c12e5d44247faae5e18100b62e760 P e4ed3778cddd4810939b05452c64f23d: Bootstrap starting.
I20260924 23:19:44.110733 18511 tablet_bootstrap.cc:654] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:44.110877 18510 tablet_bootstrap.cc:492] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597: No bootstrap required, opened a new log
I20260924 23:19:44.110944 18510 ts_tablet_manager.cc:1397] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:19:44.111003 18512 tablet_bootstrap.cc:654] T 790c12e5d44247faae5e18100b62e760 P e4ed3778cddd4810939b05452c64f23d: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:44.111233 18511 log.cc:824] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:44.111536 18512 log.cc:824] T 790c12e5d44247faae5e18100b62e760 P e4ed3778cddd4810939b05452c64f23d: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:44.111922 18511 tablet_bootstrap.cc:492] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad: No bootstrap required, opened a new log
I20260924 23:19:44.111980 18511 ts_tablet_manager.cc:1397] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:19:44.112317 18512 tablet_bootstrap.cc:492] T 790c12e5d44247faae5e18100b62e760 P e4ed3778cddd4810939b05452c64f23d: No bootstrap required, opened a new log
I20260924 23:19:44.112417 18512 ts_tablet_manager.cc:1397] T 790c12e5d44247faae5e18100b62e760 P e4ed3778cddd4810939b05452c64f23d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:19:44.112694 18510 raft_consensus.cc:348] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ed3778cddd4810939b05452c64f23d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 39315 } } peers { permanent_uuid: "03e70c3711ba4f07949f36f6d9495597" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45557 } } peers { permanent_uuid: "6eaf1133435247348a15fe19ac65a7ad" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 41303 } }
I20260924 23:19:44.112876 18510 raft_consensus.cc:374] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:44.112923 18510 raft_consensus.cc:729] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e70c3711ba4f07949f36f6d9495597, State: Initialized, Role: FOLLOWER
I20260924 23:19:44.113039 18510 consensus_queue.cc:260] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ed3778cddd4810939b05452c64f23d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 39315 } } peers { permanent_uuid: "03e70c3711ba4f07949f36f6d9495597" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45557 } } peers { permanent_uuid: "6eaf1133435247348a15fe19ac65a7ad" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 41303 } }
I20260924 23:19:44.113306 18510 ts_tablet_manager.cc:1428] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:19:44.113425 18359 heartbeater.cc:503] Master 127.1.4.254:44863 was elected leader, sending a full tablet report...
I20260924 23:19:44.113795 18511 raft_consensus.cc:348] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ed3778cddd4810939b05452c64f23d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 39315 } } peers { permanent_uuid: "03e70c3711ba4f07949f36f6d9495597" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45557 } } peers { permanent_uuid: "6eaf1133435247348a15fe19ac65a7ad" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 41303 } }
I20260924 23:19:44.113967 18511 raft_consensus.cc:374] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:44.113999 18511 raft_consensus.cc:729] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eaf1133435247348a15fe19ac65a7ad, State: Initialized, Role: FOLLOWER
I20260924 23:19:44.114130 18511 consensus_queue.cc:260] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ed3778cddd4810939b05452c64f23d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 39315 } } peers { permanent_uuid: "03e70c3711ba4f07949f36f6d9495597" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45557 } } peers { permanent_uuid: "6eaf1133435247348a15fe19ac65a7ad" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 41303 } }
I20260924 23:19:44.114389 18511 ts_tablet_manager.cc:1428] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:19:44.114411 18493 heartbeater.cc:503] Master 127.1.4.254:44863 was elected leader, sending a full tablet report...
I20260924 23:19:44.114449 18512 raft_consensus.cc:348] T 790c12e5d44247faae5e18100b62e760 P e4ed3778cddd4810939b05452c64f23d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ed3778cddd4810939b05452c64f23d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 39315 } } peers { permanent_uuid: "03e70c3711ba4f07949f36f6d9495597" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45557 } } peers { permanent_uuid: "6eaf1133435247348a15fe19ac65a7ad" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 41303 } }
I20260924 23:19:44.114602 18512 raft_consensus.cc:374] T 790c12e5d44247faae5e18100b62e760 P e4ed3778cddd4810939b05452c64f23d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:44.114637 18512 raft_consensus.cc:729] T 790c12e5d44247faae5e18100b62e760 P e4ed3778cddd4810939b05452c64f23d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ed3778cddd4810939b05452c64f23d, State: Initialized, Role: FOLLOWER
I20260924 23:19:44.114723 18512 consensus_queue.cc:260] T 790c12e5d44247faae5e18100b62e760 P e4ed3778cddd4810939b05452c64f23d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ed3778cddd4810939b05452c64f23d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 39315 } } peers { permanent_uuid: "03e70c3711ba4f07949f36f6d9495597" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45557 } } peers { permanent_uuid: "6eaf1133435247348a15fe19ac65a7ad" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 41303 } }
I20260924 23:19:44.114979 18512 ts_tablet_manager.cc:1428] T 790c12e5d44247faae5e18100b62e760 P e4ed3778cddd4810939b05452c64f23d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:19:44.115010 18225 heartbeater.cc:503] Master 127.1.4.254:44863 was elected leader, sending a full tablet report...
W20260924 23:19:44.164269 18360 tablet.cc:2369] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:19:44.180773 18517 raft_consensus.cc:482] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:19:44.180932 18517 raft_consensus.cc:504] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ed3778cddd4810939b05452c64f23d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 39315 } } peers { permanent_uuid: "03e70c3711ba4f07949f36f6d9495597" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45557 } } peers { permanent_uuid: "6eaf1133435247348a15fe19ac65a7ad" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 41303 } }
I20260924 23:19:44.181290 18517 leader_election.cc:288] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4ed3778cddd4810939b05452c64f23d (127.1.4.193:39315), 03e70c3711ba4f07949f36f6d9495597 (127.1.4.194:45557)
I20260924 23:19:44.184526 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.181652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:36127 (local address 127.1.4.194:45557)
0924 23:19:44.181690 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.181692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.181703 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:19:44.181792 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.181793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.181808 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.181875 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:44.182235 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.182752 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.183812 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.184204 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.184266 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.184278 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.184414 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.184450 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:19:44.184540 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.181640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:35873 (local address 127.1.4.193:39315)
0924 23:19:44.181688 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.181690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.181928 (+ 238us) server_negotiation.cc:407] Connection header received
0924 23:19:44.181960 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.181961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.181972 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.182025 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:44.182221 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.182706 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.183840 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.184217 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.184280 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.184299 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.184432 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.184475 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:19:44.184537 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.181533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:45557 (local address 127.1.4.195:36127)
0924 23:19:44.181627 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:44.181641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:44.181703 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:44.182082 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:44.182083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:44.182085 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:44.182142 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.182143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.182843 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.182844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.183722 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.183724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.184284 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.184285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.184346 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.184349 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:19:44.184393 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:19:44.184443 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":41}
I20260924 23:19:44.184552 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.181587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:39315 (local address 127.1.4.195:35873)
0924 23:19:44.181778 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:44.181812 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:44.181873 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:44.182044 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:44.182045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:44.182047 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:44.182130 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.182132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.182815 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.182815 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.183742 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.183744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.184297 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.184298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.184363 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.184367 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:19:44.184411 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:19:44.184451 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":70,"thread_start_us":131,"threads_started":1}
I20260924 23:19:44.185074 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "790c12e5d44247faae5e18100b62e760" candidate_uuid: "6eaf1133435247348a15fe19ac65a7ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e70c3711ba4f07949f36f6d9495597" is_pre_election: true
I20260924 23:19:44.185245 18313 raft_consensus.cc:2454] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6eaf1133435247348a15fe19ac65a7ad in term 0.
I20260924 23:19:44.185238 18179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "790c12e5d44247faae5e18100b62e760" candidate_uuid: "6eaf1133435247348a15fe19ac65a7ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ed3778cddd4810939b05452c64f23d" is_pre_election: true
I20260924 23:19:44.185413 18179 raft_consensus.cc:2454] T 790c12e5d44247faae5e18100b62e760 P e4ed3778cddd4810939b05452c64f23d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6eaf1133435247348a15fe19ac65a7ad in term 0.
I20260924 23:19:44.185503 18382 leader_election.cc:302] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e70c3711ba4f07949f36f6d9495597, 6eaf1133435247348a15fe19ac65a7ad; no voters:
I20260924 23:19:44.185635 18517 raft_consensus.cc:2781] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:19:44.185693 18517 raft_consensus.cc:482] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:19:44.185709 18517 raft_consensus.cc:3037] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:44.186460 18517 raft_consensus.cc:504] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ed3778cddd4810939b05452c64f23d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 39315 } } peers { permanent_uuid: "03e70c3711ba4f07949f36f6d9495597" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45557 } } peers { permanent_uuid: "6eaf1133435247348a15fe19ac65a7ad" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 41303 } }
I20260924 23:19:44.186596 18517 leader_election.cc:288] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [CANDIDATE]: Term 1 election: Requested vote from peers e4ed3778cddd4810939b05452c64f23d (127.1.4.193:39315), 03e70c3711ba4f07949f36f6d9495597 (127.1.4.194:45557)
I20260924 23:19:44.186801 18313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "790c12e5d44247faae5e18100b62e760" candidate_uuid: "6eaf1133435247348a15fe19ac65a7ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e70c3711ba4f07949f36f6d9495597"
I20260924 23:19:44.186872 18179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "790c12e5d44247faae5e18100b62e760" candidate_uuid: "6eaf1133435247348a15fe19ac65a7ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ed3778cddd4810939b05452c64f23d"
I20260924 23:19:44.186894 18313 raft_consensus.cc:3037] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:44.186964 18179 raft_consensus.cc:3037] T 790c12e5d44247faae5e18100b62e760 P e4ed3778cddd4810939b05452c64f23d [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:44.187696 18313 raft_consensus.cc:2454] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6eaf1133435247348a15fe19ac65a7ad in term 1.
I20260924 23:19:44.187767 18179 raft_consensus.cc:2454] T 790c12e5d44247faae5e18100b62e760 P e4ed3778cddd4810939b05452c64f23d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6eaf1133435247348a15fe19ac65a7ad in term 1.
I20260924 23:19:44.187857 18382 leader_election.cc:302] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e70c3711ba4f07949f36f6d9495597, 6eaf1133435247348a15fe19ac65a7ad; no voters:
I20260924 23:19:44.187951 18517 raft_consensus.cc:2781] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:44.188112 18517 raft_consensus.cc:686] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [term 1 LEADER]: Becoming Leader. State: Replica: 6eaf1133435247348a15fe19ac65a7ad, State: Running, Role: LEADER
I20260924 23:19:44.188205 18517 consensus_queue.cc:237] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ed3778cddd4810939b05452c64f23d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 39315 } } peers { permanent_uuid: "03e70c3711ba4f07949f36f6d9495597" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45557 } } peers { permanent_uuid: "6eaf1133435247348a15fe19ac65a7ad" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 41303 } }
I20260924 23:19:44.189098 18034 catalog_manager.cc:5495] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 6eaf1133435247348a15fe19ac65a7ad (127.1.4.195). New cstate: current_term: 1 leader_uuid: "6eaf1133435247348a15fe19ac65a7ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ed3778cddd4810939b05452c64f23d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 39315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03e70c3711ba4f07949f36f6d9495597" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6eaf1133435247348a15fe19ac65a7ad" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 41303 } health_report { overall_health: HEALTHY } } }
I20260924 23:19:44.200146 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.197805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44126 (local address 127.1.4.195:41303)
0924 23:19:44.197851 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.197853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.197861 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:19:44.197915 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.197916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.197935 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.197994 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:19:44.198089 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.198545 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.199430 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.199636 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.199687 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.199874 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:19:44.199979 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:19:44.200038 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.200056 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.200074 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:19:44.201964 18179 raft_consensus.cc:1261] T 790c12e5d44247faae5e18100b62e760 P e4ed3778cddd4810939b05452c64f23d [term 1 FOLLOWER]: Refusing update from remote peer 6eaf1133435247348a15fe19ac65a7ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:19:44.201967 18313 raft_consensus.cc:1261] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597 [term 1 FOLLOWER]: Refusing update from remote peer 6eaf1133435247348a15fe19ac65a7ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:19:44.202332 18520 consensus_queue.cc:1035] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ed3778cddd4810939b05452c64f23d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 39315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:19:44.202464 18517 consensus_queue.cc:1035] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "03e70c3711ba4f07949f36f6d9495597" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:19:44.204754 18523 mvcc.cc:205] Tried to move back new op lower bound from 7333035967288643584 to 7333035967235641344. Current Snapshot: MvccSnapshot[applied={T|T < 7333035967288643584}]
I20260924 23:19:44.204918 18524 mvcc.cc:205] Tried to move back new op lower bound from 7333035967288643584 to 7333035967235641344. Current Snapshot: MvccSnapshot[applied={T|T < 7333035967288643584}]
I20260924 23:19:44.205610 18525 mvcc.cc:205] Tried to move back new op lower bound from 7333035967288643584 to 7333035967235641344. Current Snapshot: MvccSnapshot[applied={T|T < 7333035967288643584}]
I20260924 23:19:44.207906 18033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:19:44.207991 18033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:19:44.208762 18033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5367d19c7c6a409daa58c453afd7bcfc: Sending DeleteTablet for 3 replicas of tablet 790c12e5d44247faae5e18100b62e760
I20260924 23:19:44.209182 18159 tablet_service.cc:1514] Processing DeleteTablet for tablet 790c12e5d44247faae5e18100b62e760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:44 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:43736
I20260924 23:19:44.209174 18293 tablet_service.cc:1514] Processing DeleteTablet for tablet 790c12e5d44247faae5e18100b62e760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:44 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:40034
I20260924 23:19:44.209198 18427 tablet_service.cc:1514] Processing DeleteTablet for tablet 790c12e5d44247faae5e18100b62e760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:44 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:44122
I20260924 23:19:44.209434 18534 tablet_replica.cc:330] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad: stopping tablet replica
I20260924 23:19:44.209532 18534 raft_consensus.cc:2229] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:19:44.209654 18534 raft_consensus.cc:2258] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:19:44.209652 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 18096
I20260924 23:19:44.212821 18534 ts_tablet_manager.cc:1905] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:19:44.214412 18534 ts_tablet_manager.cc:1918] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:19:44.214488 18534 log.cc:1196] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/790c12e5d44247faae5e18100b62e760
I20260924 23:19:44.214764 18534 ts_tablet_manager.cc:1939] T 790c12e5d44247faae5e18100b62e760 P 6eaf1133435247348a15fe19ac65a7ad: Deleting consensus metadata
I20260924 23:19:44.215167 18021 catalog_manager.cc:4841] TS 6eaf1133435247348a15fe19ac65a7ad (127.1.4.195:41303): tablet 790c12e5d44247faae5e18100b62e760 (table test-table [id=6880b622b54d4e378ab13360d9ef8b65]) successfully deleted
I20260924 23:19:44.215503 18532 tablet_replica.cc:330] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597: stopping tablet replica
I20260924 23:19:44.215631 18532 raft_consensus.cc:2229] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:19:44.215744 18532 raft_consensus.cc:2258] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:19:44.216185 18532 ts_tablet_manager.cc:1905] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:19:44.217926 18532 ts_tablet_manager.cc:1918] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:19:44.218003 18532 log.cc:1196] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/790c12e5d44247faae5e18100b62e760
W20260924 23:19:44.218181 18019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.4.193:39315 (error 108)
I20260924 23:19:44.218272 18532 ts_tablet_manager.cc:1939] T 790c12e5d44247faae5e18100b62e760 P 03e70c3711ba4f07949f36f6d9495597: Deleting consensus metadata
I20260924 23:19:44.218721 18020 catalog_manager.cc:4841] TS 03e70c3711ba4f07949f36f6d9495597 (127.1.4.194:45557): tablet 790c12e5d44247faae5e18100b62e760 (table test-table [id=6880b622b54d4e378ab13360d9ef8b65]) successfully deleted
I20260924 23:19:44.218788 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.218623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:39315 (local address 127.0.0.1:43744)
0924 23:19:44.218665 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:44.218708 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:39315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 23:19:44.218909 18019 catalog_manager.cc:4584] TS e4ed3778cddd4810939b05452c64f23d (127.1.4.193:39315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 790c12e5d44247faae5e18100b62e760: Network error: Client connection negotiation failed: client connection to 127.1.4.193:39315: connect: Connection refused (error 111)
I20260924 23:19:44.219527 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 18230
I20260924 23:19:44.226751 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 18364
I20260924 23:19:44.230983 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 18002
2026-09-24T23:19:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1856 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T23:19:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:19:44Z Disabled control of system clock
I20260924 23:19:44.256621 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:45245
--webserver_interface=127.1.4.254
--webserver_port=0
--builtin_ntp_servers=127.1.4.212:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:45245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 23:19:44.337250 18539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:44.337467 18539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:44.337492 18539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:44.338616 18539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:19:44.338657 18539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:44.338670 18539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:44.338680 18539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:19:44.338690 18539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:19:44.340025 18539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:45245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:45245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:44.340237 18539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:44.340507 18539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:44.345341 18544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:44.345355 18547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:44.345340 18545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:44.345374 18539 server_base.cc:1034] running on GCE node
I20260924 23:19:44.345806 18539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:44.346105 18539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:44.347266 18539 hybrid_clock.cc:648] HybridClock initialized: now 1790291984347244 us; error 35 us; skew 500 ppm
I20260924 23:19:44.348484 18539 webserver.cc:460] Webserver started at http://127.1.4.254:46553/ using document root <none> and password file <none>
I20260924 23:19:44.348718 18539 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:44.348800 18539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:44.348927 18539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:44.349972 18539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "5092e10ae42c492fb8559bda412248ed"
format_stamp: "Formatted at 2026-09-24 23:19:44 on dist-test-slave-s1hw"
server_key: "d8c58d77b6f566d9b1a3222c1518070d"
server_key_iv: "3df417b05483c936f36a2df678d32d58"
server_key_version: "encryptionkey@0"
I20260924 23:19:44.350317 18539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "5092e10ae42c492fb8559bda412248ed"
format_stamp: "Formatted at 2026-09-24 23:19:44 on dist-test-slave-s1hw"
server_key: "d8c58d77b6f566d9b1a3222c1518070d"
server_key_iv: "3df417b05483c936f36a2df678d32d58"
server_key_version: "encryptionkey@0"
I20260924 23:19:44.351553 18539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:19:44.352228 18553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:44.352380 18539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:44.352460 18539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "5092e10ae42c492fb8559bda412248ed"
format_stamp: "Formatted at 2026-09-24 23:19:44 on dist-test-slave-s1hw"
server_key: "d8c58d77b6f566d9b1a3222c1518070d"
server_key_iv: "3df417b05483c936f36a2df678d32d58"
server_key_version: "encryptionkey@0"
I20260924 23:19:44.352527 18539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:44.371405 18539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:44.371747 18539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:44.371881 18539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:44.375617 18539 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:45245
I20260924 23:19:44.375654 18605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:45245 every 8 connection(s)
I20260924 23:19:44.376001 18539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:19:44.376983 18606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:44.379254 18606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed: Bootstrap starting.
I20260924 23:19:44.379839 18606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:44.380080 18606 log.cc:824] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:44.380700 18606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed: No bootstrap required, opened a new log
I20260924 23:19:44.380887 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 18539
I20260924 23:19:44.380964 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:19:44.381137 1043 external_mini_cluster.cc:1428] Setting key f2efa75d9cdf4cf39b8908063f322d27
I20260924 23:19:44.382826 18606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5092e10ae42c492fb8559bda412248ed" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45245 } }
I20260924 23:19:44.383010 18606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:44.383046 18606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5092e10ae42c492fb8559bda412248ed, State: Initialized, Role: FOLLOWER
I20260924 23:19:44.383206 18606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5092e10ae42c492fb8559bda412248ed" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45245 } }
I20260924 23:19:44.383296 18606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:19:44.383327 18606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:19:44.383394 18606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:44.384119 18606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5092e10ae42c492fb8559bda412248ed" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45245 } }
I20260924 23:19:44.384239 18606 leader_election.cc:302] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5092e10ae42c492fb8559bda412248ed; no voters:
I20260924 23:19:44.384408 18606 leader_election.cc:288] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:19:44.384466 18611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:44.384650 18611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [term 1 LEADER]: Becoming Leader. State: Replica: 5092e10ae42c492fb8559bda412248ed, State: Running, Role: LEADER
I20260924 23:19:44.384657 18606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:19:44.384811 18611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5092e10ae42c492fb8559bda412248ed" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45245 } }
I20260924 23:19:44.385254 18612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5092e10ae42c492fb8559bda412248ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5092e10ae42c492fb8559bda412248ed" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45245 } } }
I20260924 23:19:44.385337 18612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:44.385516 18613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5092e10ae42c492fb8559bda412248ed. Latest consensus state: current_term: 1 leader_uuid: "5092e10ae42c492fb8559bda412248ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5092e10ae42c492fb8559bda412248ed" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45245 } } }
I20260924 23:19:44.385583 18613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed [sys.catalog]: This master's current role is: LEADER
W20260924 23:19:44.386570 18627 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:19:44.386618 18627 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:19:44.386663 18619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:19:44.386801 18619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:19:44.388411 18619 catalog_manager.cc:1347] Generated new cluster ID: 3be57eed27eb4e9d929dfddd26386efc
I20260924 23:19:44.388468 18619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:19:44.388885 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.381680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51694 (local address 127.1.4.254:45245)
0924 23:19:44.381917 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.381926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.381959 (+ 33us) server_negotiation.cc:407] Connection header received
0924 23:19:44.382766 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.382775 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.382864 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.383020 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:44.383271 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.385010 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.386174 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.386843 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.386965 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.387937 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:44.387941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:44.387941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:44.388142 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:44.388193 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:44.388209 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:44.388280 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.388570 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.388660 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260924 23:19:44.410554 18619 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:19:44.411317 18619 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:19:44.429615 18619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed: Generated new TSK 0
I20260924 23:19:44.429841 18619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:19:44.440317 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45245
--builtin_ntp_servers=127.1.4.212:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 23:19:44.521277 18630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:44.521494 18630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:44.521515 18630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:44.522686 18630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:44.522739 18630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:44.522766 18630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:19:44.524122 18630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:44.524377 18630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:44.524672 18630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:44.529657 18638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:44.529712 18636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:44.529671 18635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:44.529884 18630 server_base.cc:1034] running on GCE node
I20260924 23:19:44.530112 18630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:44.530362 18630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:44.531502 18630 hybrid_clock.cc:648] HybridClock initialized: now 1790291984531484 us; error 33 us; skew 500 ppm
I20260924 23:19:44.532692 18630 webserver.cc:460] Webserver started at http://127.1.4.193:35027/ using document root <none> and password file <none>
I20260924 23:19:44.532932 18630 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:44.532981 18630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:44.533109 18630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:44.534155 18630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "60871e8ec112431f84c5183cef800239"
format_stamp: "Formatted at 2026-09-24 23:19:44 on dist-test-slave-s1hw"
server_key: "2fe1dd33bef8c8a8015c48ad7fc9b7fe"
server_key_iv: "721af64e043e979ecb355e9665a129a6"
server_key_version: "encryptionkey@0"
I20260924 23:19:44.534480 18630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "60871e8ec112431f84c5183cef800239"
format_stamp: "Formatted at 2026-09-24 23:19:44 on dist-test-slave-s1hw"
server_key: "2fe1dd33bef8c8a8015c48ad7fc9b7fe"
server_key_iv: "721af64e043e979ecb355e9665a129a6"
server_key_version: "encryptionkey@0"
I20260924 23:19:44.535766 18630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:19:44.536476 18644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:44.536639 18630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:19:44.536720 18630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "60871e8ec112431f84c5183cef800239"
format_stamp: "Formatted at 2026-09-24 23:19:44 on dist-test-slave-s1hw"
server_key: "2fe1dd33bef8c8a8015c48ad7fc9b7fe"
server_key_iv: "721af64e043e979ecb355e9665a129a6"
server_key_version: "encryptionkey@0"
I20260924 23:19:44.536787 18630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:44.544497 18630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:44.544787 18630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:44.544906 18630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:44.545109 18630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:44.545400 18630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:44.545433 18630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:44.545461 18630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:44.545487 18630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:44.550904 18630 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:33125
I20260924 23:19:44.550959 18757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:33125 every 8 connection(s)
I20260924 23:19:44.551262 18630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:19:44.555225 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 18630
I20260924 23:19:44.555321 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:19:44.555536 1043 external_mini_cluster.cc:1428] Setting key 05cbf71994d2e2822b76628755e39dd4
I20260924 23:19:44.556470 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.551964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:38903 (local address 127.1.4.254:45245)
0924 23:19:44.552030 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.552032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.552265 (+ 233us) server_negotiation.cc:407] Connection header received
0924 23:19:44.552757 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.552758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.552770 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.552830 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:44.553481 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.553980 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.554904 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.555117 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.555190 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.556169 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:44.556170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:44.556171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:44.556176 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:44.556192 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:44.556196 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:44.556246 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.556361 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.556391 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 23:19:44.556602 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45245
--builtin_ntp_servers=127.1.4.212:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 23:19:44.556667 18760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.551940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:45245 (local address 127.1.4.193:38903)
0924 23:19:44.552141 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:44.552182 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:44.552658 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:44.552899 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:44.552902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:44.552911 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:44.553416 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.553420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.554049 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.554049 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.554833 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.554836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.555218 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.555219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.555440 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.556029 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:19:44.556029 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:44.556058 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:19:44.556081 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:44.556249 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:44.556254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:44.556258 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:19:44.556358 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 23:19:44.556445 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260924 23:19:44.557091 18758 heartbeater.cc:347] Connected to a master server at 127.1.4.254:45245
I20260924 23:19:44.557190 18758 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:44.557374 18758 heartbeater.cc:511] Master 127.1.4.254:45245 requested a full tablet report, sending...
I20260924 23:19:44.557910 18570 ts_manager.cc:194] Registered new tserver with Master: 60871e8ec112431f84c5183cef800239 (127.1.4.193:33125)
I20260924 23:19:44.558528 18570 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.4.193:38903
W20260924 23:19:44.637220 18761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:44.637434 18761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:44.637465 18761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:44.638617 18761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:44.638655 18761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:44.638688 18761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:19:44.640126 18761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:44.640367 18761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:44.640657 18761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:44.645583 18766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:44.645722 18767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:44.645583 18769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:44.645885 18761 server_base.cc:1034] running on GCE node
I20260924 23:19:44.646099 18761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:44.646349 18761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:44.647483 18761 hybrid_clock.cc:648] HybridClock initialized: now 1790291984647468 us; error 23 us; skew 500 ppm
I20260924 23:19:44.648641 18761 webserver.cc:460] Webserver started at http://127.1.4.194:42001/ using document root <none> and password file <none>
I20260924 23:19:44.648908 18761 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:44.648958 18761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:44.649084 18761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:44.650166 18761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "286b0cfe82484b7691977229f5d54bc8"
format_stamp: "Formatted at 2026-09-24 23:19:44 on dist-test-slave-s1hw"
server_key: "283294deee43c1e229b44e8a84039e94"
server_key_iv: "d563ac3445432dc05b5bbcf629de0b63"
server_key_version: "encryptionkey@0"
I20260924 23:19:44.650518 18761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "286b0cfe82484b7691977229f5d54bc8"
format_stamp: "Formatted at 2026-09-24 23:19:44 on dist-test-slave-s1hw"
server_key: "283294deee43c1e229b44e8a84039e94"
server_key_iv: "d563ac3445432dc05b5bbcf629de0b63"
server_key_version: "encryptionkey@0"
I20260924 23:19:44.651732 18761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:19:44.652374 18775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:44.652523 18761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:44.652602 18761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "286b0cfe82484b7691977229f5d54bc8"
format_stamp: "Formatted at 2026-09-24 23:19:44 on dist-test-slave-s1hw"
server_key: "283294deee43c1e229b44e8a84039e94"
server_key_iv: "d563ac3445432dc05b5bbcf629de0b63"
server_key_version: "encryptionkey@0"
I20260924 23:19:44.652661 18761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:44.659662 18761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:44.659930 18761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:44.660048 18761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:44.660235 18761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:44.660519 18761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:44.660550 18761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:44.660578 18761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:44.660604 18761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:44.666023 18761 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:40789
I20260924 23:19:44.666069 18888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:40789 every 8 connection(s)
I20260924 23:19:44.666375 18761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:19:44.671051 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 18761
I20260924 23:19:44.671165 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:19:44.671294 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.667111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:47979 (local address 127.1.4.254:45245)
0924 23:19:44.667167 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.667169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.667408 (+ 239us) server_negotiation.cc:407] Connection header received
0924 23:19:44.667905 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.667906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.667917 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.667980 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:44.668592 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.669098 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.670043 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.670228 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.670275 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.671020 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:44.671022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:44.671023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:44.671028 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:44.671046 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:44.671050 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:44.671111 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.671186 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.671217 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 23:19:44.671411 1043 external_mini_cluster.cc:1428] Setting key 0218bef4c469ebc8039e64a0ae29b4be
I20260924 23:19:44.671646 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.667085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:45245 (local address 127.1.4.194:47979)
0924 23:19:44.667250 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:44.667290 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:44.667819 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:44.668035 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:44.668038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:44.668047 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:44.668525 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.668530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.669170 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.669170 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.669977 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.669980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.670302 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.670304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.670406 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.670875 (+ 469us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:19:44.670876 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:44.670906 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:19:44.670931 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:44.671113 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:44.671116 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:44.671118 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:19:44.671180 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 23:19:44.671287 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260924 23:19:44.672144 18889 heartbeater.cc:347] Connected to a master server at 127.1.4.254:45245
I20260924 23:19:44.672230 18889 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:44.672308 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45245
--builtin_ntp_servers=127.1.4.212:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 23:19:44.672405 18889 heartbeater.cc:511] Master 127.1.4.254:45245 requested a full tablet report, sending...
I20260924 23:19:44.672797 18570 ts_manager.cc:194] Registered new tserver with Master: 286b0cfe82484b7691977229f5d54bc8 (127.1.4.194:40789)
I20260924 23:19:44.673238 18570 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.4.194:47979
W20260924 23:19:44.751389 18892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:44.751621 18892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:44.751643 18892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:44.752802 18892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:44.752839 18892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:44.752864 18892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:19:44.754184 18892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:44.754395 18892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:44.754704 18892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:44.759897 18897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:44.759912 18892 server_base.cc:1034] running on GCE node
W20260924 23:19:44.759862 18898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:44.759913 18900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:44.760180 18892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:44.760412 18892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:44.761543 18892 hybrid_clock.cc:648] HybridClock initialized: now 1790291984761531 us; error 30 us; skew 500 ppm
I20260924 23:19:44.762866 18892 webserver.cc:460] Webserver started at http://127.1.4.195:39899/ using document root <none> and password file <none>
I20260924 23:19:44.763099 18892 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:44.763145 18892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:44.763269 18892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:44.764390 18892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "6cfa957cb5914007b99df2032731a574"
format_stamp: "Formatted at 2026-09-24 23:19:44 on dist-test-slave-s1hw"
server_key: "96fdc24ebfdf67af52c82c512fdaefbe"
server_key_iv: "895874ff7a134137722f6bfe3da7ec9f"
server_key_version: "encryptionkey@0"
I20260924 23:19:44.764704 18892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "6cfa957cb5914007b99df2032731a574"
format_stamp: "Formatted at 2026-09-24 23:19:44 on dist-test-slave-s1hw"
server_key: "96fdc24ebfdf67af52c82c512fdaefbe"
server_key_iv: "895874ff7a134137722f6bfe3da7ec9f"
server_key_version: "encryptionkey@0"
I20260924 23:19:44.766041 18892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:19:44.766832 18906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:44.766995 18892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:19:44.767072 18892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "6cfa957cb5914007b99df2032731a574"
format_stamp: "Formatted at 2026-09-24 23:19:44 on dist-test-slave-s1hw"
server_key: "96fdc24ebfdf67af52c82c512fdaefbe"
server_key_iv: "895874ff7a134137722f6bfe3da7ec9f"
server_key_version: "encryptionkey@0"
I20260924 23:19:44.767138 18892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:44.783324 18892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:44.783737 18892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:44.783881 18892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:44.784116 18892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:44.784443 18892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:44.784477 18892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:44.784507 18892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:44.784523 18892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:44.790257 18892 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:37737
I20260924 23:19:44.790325 19019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:37737 every 8 connection(s)
I20260924 23:19:44.790650 18892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:19:44.795706 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.791415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:39583 (local address 127.1.4.254:45245)
0924 23:19:44.791477 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.791479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.791724 (+ 245us) server_negotiation.cc:407] Connection header received
0924 23:19:44.792218 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.792220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.792230 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.792291 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:44.792993 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.793487 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.794455 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.794642 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.794689 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.795469 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:44.795470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:44.795470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:44.795474 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:44.795489 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:44.795493 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:44.795532 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.795600 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.795627 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 23:19:44.795923 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.791397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:45245 (local address 127.1.4.195:39583)
0924 23:19:44.791585 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:44.791627 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:44.792098 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:44.792359 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:44.792363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:44.792371 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:44.792920 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.792925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.793550 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.793550 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.794382 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.794384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.794744 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.794745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.794855 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.795318 (+ 463us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:19:44.795319 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:44.795347 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:19:44.795386 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:44.795541 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:44.795544 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:44.795546 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:19:44.795593 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 23:19:44.795669 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260924 23:19:44.796363 19020 heartbeater.cc:347] Connected to a master server at 127.1.4.254:45245
I20260924 23:19:44.796443 19020 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:44.796615 19020 heartbeater.cc:511] Master 127.1.4.254:45245 requested a full tablet report, sending...
I20260924 23:19:44.796909 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 18892
I20260924 23:19:44.797004 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:19:44.797015 18570 ts_manager.cc:194] Registered new tserver with Master: 6cfa957cb5914007b99df2032731a574 (127.1.4.195:37737)
I20260924 23:19:44.797214 1043 external_mini_cluster.cc:1428] Setting key bcd7e86495f54d8578e2067b05f0c594
I20260924 23:19:44.797501 18570 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.4.195:39583
I20260924 23:19:44.798178 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:19:44.802263 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.799406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51710 (local address 127.1.4.254:45245)
0924 23:19:44.799451 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.799453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.799610 (+ 157us) server_negotiation.cc:407] Connection header received
0924 23:19:44.799665 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.799665 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.799677 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.799718 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:44.799869 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.800349 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.801092 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.801292 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.801345 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.802032 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:44.802033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:44.802033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:44.802038 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:44.802053 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:44.802057 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:44.802113 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.802165 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.802188 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:19:44.803742 18570 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:19:44.805344 18570 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:19:44.811581 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.807490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:33125 (local address 127.0.0.1:58494)
0924 23:19:44.807661 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:44.807706 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:44.807775 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:44.808416 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:44.808417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:44.808419 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:44.808464 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.808465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.809726 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.809727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.810583 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.810585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.811255 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.811256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.811342 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.811357 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 23:19:44.811446 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 23:19:44.811504 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":58,"threads_started":1}
I20260924 23:19:44.811591 19030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.807590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58494 (local address 127.1.4.193:33125)
0924 23:19:44.808199 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.808204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.808212 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:19:44.808268 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.808270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.808283 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.808351 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:44.808543 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.809647 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.810657 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.811163 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.811226 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.811321 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.811471 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.811510 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":85,"threads_started":1}
I20260924 23:19:44.811808 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.807684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:37737 (local address 127.0.0.1:43642)
0924 23:19:44.808023 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:44.808032 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:44.808080 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:44.808311 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:44.808312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:44.808314 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:44.808357 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.808359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.809308 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.809309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.810585 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.810587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.811616 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.811617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.811667 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.811669 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:19:44.811709 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 23:19:44.811740 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260924 23:19:44.811981 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.807499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:40789 (local address 127.0.0.1:41120)
0924 23:19:44.807780 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:44.807800 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:44.807853 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:44.808064 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:44.808067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:44.808069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:44.808135 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.808137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.809551 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.809552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.810623 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.810624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.811530 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.811531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.811591 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.811596 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:19:44.811625 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 23:19:44.811666 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":23,"thread_start_us":111,"threads_started":1}
I20260924 23:19:44.812083 19033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.807866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.1.4.195:37737)
0924 23:19:44.808006 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.808010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.808070 (+ 60us) server_negotiation.cc:407] Connection header received
0924 23:19:44.808160 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.808163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.808178 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.808246 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:44.808425 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.809230 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.810687 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.811436 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.811501 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.811589 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.811966 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.812005 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260924 23:19:44.812386 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.807672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.1.4.194:40789)
0924 23:19:44.807926 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.807930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.807938 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:19:44.807969 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.807970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.807982 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.808047 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:44.808249 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.809469 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.810700 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.811437 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.812146 (+ 709us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.812249 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.812272 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.812308 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":146,"threads_started":1}
I20260924 23:19:44.812456 18692 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce1ffbf5beb467daed30fa930220b65 (DEFAULT_TABLE table=test-table [id=e535570bf77a4268b2173e73e07426b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:44.812670 18954 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce1ffbf5beb467daed30fa930220b65 (DEFAULT_TABLE table=test-table [id=e535570bf77a4268b2173e73e07426b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:44.812834 18692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce1ffbf5beb467daed30fa930220b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:44.812952 18954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce1ffbf5beb467daed30fa930220b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:44.812922 18823 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce1ffbf5beb467daed30fa930220b65 (DEFAULT_TABLE table=test-table [id=e535570bf77a4268b2173e73e07426b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:44.813158 18823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce1ffbf5beb467daed30fa930220b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:44.815501 19034 tablet_bootstrap.cc:492] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8: Bootstrap starting.
I20260924 23:19:44.816159 19034 tablet_bootstrap.cc:654] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:44.816429 19034 log.cc:824] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:44.816485 19036 tablet_bootstrap.cc:492] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574: Bootstrap starting.
I20260924 23:19:44.816578 19035 tablet_bootstrap.cc:492] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239: Bootstrap starting.
I20260924 23:19:44.817108 19036 tablet_bootstrap.cc:654] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:44.817108 19035 tablet_bootstrap.cc:654] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:44.817410 19036 log.cc:824] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:44.817410 19035 log.cc:824] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:44.818104 19036 tablet_bootstrap.cc:492] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574: No bootstrap required, opened a new log
I20260924 23:19:44.818125 19034 tablet_bootstrap.cc:492] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8: No bootstrap required, opened a new log
I20260924 23:19:44.818177 19034 ts_tablet_manager.cc:1397] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 23:19:44.818177 19036 ts_tablet_manager.cc:1397] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:19:44.818297 19035 tablet_bootstrap.cc:492] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239: No bootstrap required, opened a new log
I20260924 23:19:44.818356 19035 ts_tablet_manager.cc:1397] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:19:44.820178 19034 raft_consensus.cc:348] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60871e8ec112431f84c5183cef800239" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33125 } } peers { permanent_uuid: "6cfa957cb5914007b99df2032731a574" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37737 } } peers { permanent_uuid: "286b0cfe82484b7691977229f5d54bc8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 40789 } }
I20260924 23:19:44.820325 19034 raft_consensus.cc:374] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:44.820358 19034 raft_consensus.cc:729] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286b0cfe82484b7691977229f5d54bc8, State: Initialized, Role: FOLLOWER
I20260924 23:19:44.820470 19034 consensus_queue.cc:260] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60871e8ec112431f84c5183cef800239" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33125 } } peers { permanent_uuid: "6cfa957cb5914007b99df2032731a574" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37737 } } peers { permanent_uuid: "286b0cfe82484b7691977229f5d54bc8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 40789 } }
I20260924 23:19:44.820748 19034 ts_tablet_manager.cc:1428] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:19:44.820786 18889 heartbeater.cc:503] Master 127.1.4.254:45245 was elected leader, sending a full tablet report...
I20260924 23:19:44.820708 19035 raft_consensus.cc:348] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60871e8ec112431f84c5183cef800239" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33125 } } peers { permanent_uuid: "6cfa957cb5914007b99df2032731a574" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37737 } } peers { permanent_uuid: "286b0cfe82484b7691977229f5d54bc8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 40789 } }
I20260924 23:19:44.820840 19035 raft_consensus.cc:374] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:44.820880 19035 raft_consensus.cc:729] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60871e8ec112431f84c5183cef800239, State: Initialized, Role: FOLLOWER
I20260924 23:19:44.820984 19035 consensus_queue.cc:260] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60871e8ec112431f84c5183cef800239" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33125 } } peers { permanent_uuid: "6cfa957cb5914007b99df2032731a574" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37737 } } peers { permanent_uuid: "286b0cfe82484b7691977229f5d54bc8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 40789 } }
I20260924 23:19:44.820629 19036 raft_consensus.cc:348] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60871e8ec112431f84c5183cef800239" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33125 } } peers { permanent_uuid: "6cfa957cb5914007b99df2032731a574" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37737 } } peers { permanent_uuid: "286b0cfe82484b7691977229f5d54bc8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 40789 } }
I20260924 23:19:44.821089 19036 raft_consensus.cc:374] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:44.821138 19036 raft_consensus.cc:729] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cfa957cb5914007b99df2032731a574, State: Initialized, Role: FOLLOWER
I20260924 23:19:44.821236 19035 ts_tablet_manager.cc:1428] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:19:44.821278 19036 consensus_queue.cc:260] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60871e8ec112431f84c5183cef800239" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33125 } } peers { permanent_uuid: "6cfa957cb5914007b99df2032731a574" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37737 } } peers { permanent_uuid: "286b0cfe82484b7691977229f5d54bc8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 40789 } }
I20260924 23:19:44.821359 18758 heartbeater.cc:503] Master 127.1.4.254:45245 was elected leader, sending a full tablet report...
I20260924 23:19:44.821509 19036 ts_tablet_manager.cc:1428] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:19:44.821556 19020 heartbeater.cc:503] Master 127.1.4.254:45245 was elected leader, sending a full tablet report...
W20260924 23:19:44.916956 18890 tablet.cc:2369] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:19:44.955783 19041 raft_consensus.cc:482] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:19:44.955945 19041 raft_consensus.cc:504] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60871e8ec112431f84c5183cef800239" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33125 } } peers { permanent_uuid: "6cfa957cb5914007b99df2032731a574" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37737 } } peers { permanent_uuid: "286b0cfe82484b7691977229f5d54bc8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 40789 } }
I20260924 23:19:44.956341 19041 leader_election.cc:288] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cfa957cb5914007b99df2032731a574 (127.1.4.195:37737), 286b0cfe82484b7691977229f5d54bc8 (127.1.4.194:40789)
I20260924 23:19:44.959259 18760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.956532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:37737 (local address 127.1.4.193:51501)
0924 23:19:44.956584 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:44.956597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:44.956670 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:44.956850 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:44.956851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:44.956852 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:44.956925 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.956927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.957586 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.957587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.958413 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.958416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.958987 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.958989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.959062 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.959066 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:19:44.959144 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 23:19:44.959185 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 23:19:44.959282 19033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.956602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:51501 (local address 127.1.4.195:37737)
0924 23:19:44.956659 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.956662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.956755 (+ 93us) server_negotiation.cc:407] Connection header received
0924 23:19:44.956786 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.956787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.956797 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.956848 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:44.957001 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.957492 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.958512 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.958896 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.958949 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.958957 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.959166 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.959211 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 23:19:44.959390 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.956745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:38317 (local address 127.1.4.194:40789)
0924 23:19:44.956813 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.956814 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.956822 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:19:44.956851 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.956852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.956861 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.956910 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:44.957110 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.957610 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.958604 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.959002 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.959075 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.959086 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.959262 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.959304 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 23:19:44.959399 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.956558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:40789 (local address 127.1.4.193:38317)
0924 23:19:44.956702 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:44.956711 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:44.956759 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:44.956938 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:44.956943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:44.956945 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:44.957017 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.957019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.957713 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.957714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.958496 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:44.958499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.959088 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.959089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:44.959186 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.959190 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:19:44.959243 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 23:19:44.959280 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260924 23:19:44.959722 18843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce1ffbf5beb467daed30fa930220b65" candidate_uuid: "60871e8ec112431f84c5183cef800239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "286b0cfe82484b7691977229f5d54bc8" is_pre_election: true
I20260924 23:19:44.959903 18843 raft_consensus.cc:2454] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60871e8ec112431f84c5183cef800239 in term 0.
I20260924 23:19:44.959910 18974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce1ffbf5beb467daed30fa930220b65" candidate_uuid: "60871e8ec112431f84c5183cef800239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cfa957cb5914007b99df2032731a574" is_pre_election: true
I20260924 23:19:44.960064 18974 raft_consensus.cc:2454] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60871e8ec112431f84c5183cef800239 in term 0.
I20260924 23:19:44.960148 18645 leader_election.cc:302] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 286b0cfe82484b7691977229f5d54bc8, 60871e8ec112431f84c5183cef800239; no voters:
I20260924 23:19:44.960287 19041 raft_consensus.cc:2781] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:19:44.960351 19041 raft_consensus.cc:482] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:19:44.960367 19041 raft_consensus.cc:3037] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:44.961033 19041 raft_consensus.cc:504] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60871e8ec112431f84c5183cef800239" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33125 } } peers { permanent_uuid: "6cfa957cb5914007b99df2032731a574" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37737 } } peers { permanent_uuid: "286b0cfe82484b7691977229f5d54bc8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 40789 } }
I20260924 23:19:44.961163 19041 leader_election.cc:288] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [CANDIDATE]: Term 1 election: Requested vote from peers 6cfa957cb5914007b99df2032731a574 (127.1.4.195:37737), 286b0cfe82484b7691977229f5d54bc8 (127.1.4.194:40789)
I20260924 23:19:44.961318 18974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce1ffbf5beb467daed30fa930220b65" candidate_uuid: "60871e8ec112431f84c5183cef800239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cfa957cb5914007b99df2032731a574"
I20260924 23:19:44.961390 18974 raft_consensus.cc:3037] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:44.961421 18843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce1ffbf5beb467daed30fa930220b65" candidate_uuid: "60871e8ec112431f84c5183cef800239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "286b0cfe82484b7691977229f5d54bc8"
I20260924 23:19:44.961519 18843 raft_consensus.cc:3037] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:44.962148 18974 raft_consensus.cc:2454] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60871e8ec112431f84c5183cef800239 in term 1.
I20260924 23:19:44.962258 18843 raft_consensus.cc:2454] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60871e8ec112431f84c5183cef800239 in term 1.
I20260924 23:19:44.962334 18646 leader_election.cc:302] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60871e8ec112431f84c5183cef800239, 6cfa957cb5914007b99df2032731a574; no voters:
I20260924 23:19:44.962435 19041 raft_consensus.cc:2781] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:44.962579 19041 raft_consensus.cc:686] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [term 1 LEADER]: Becoming Leader. State: Replica: 60871e8ec112431f84c5183cef800239, State: Running, Role: LEADER
I20260924 23:19:44.962762 19041 consensus_queue.cc:237] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60871e8ec112431f84c5183cef800239" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33125 } } peers { permanent_uuid: "6cfa957cb5914007b99df2032731a574" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37737 } } peers { permanent_uuid: "286b0cfe82484b7691977229f5d54bc8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 40789 } }
I20260924 23:19:44.963801 18569 catalog_manager.cc:5495] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60871e8ec112431f84c5183cef800239 (127.1.4.193). New cstate: current_term: 1 leader_uuid: "60871e8ec112431f84c5183cef800239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60871e8ec112431f84c5183cef800239" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cfa957cb5914007b99df2032731a574" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "286b0cfe82484b7691977229f5d54bc8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 40789 } health_report { overall_health: UNKNOWN } } }
I20260924 23:19:44.995990 19030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:44.993716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.1.4.193:33125)
0924 23:19:44.993773 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:44.993775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:44.993783 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:19:44.993819 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:44.993820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:44.993840 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:44.993898 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:19:44.994001 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.994495 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.995220 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:44.995427 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:44.995475 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:44.995668 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:19:44.995780 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:19:44.995854 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:44.995871 (+ 17us) server_negotiation.cc:299] Negotiation successful
0924 23:19:44.995890 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 23:19:44.997790 18974 raft_consensus.cc:1261] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574 [term 1 FOLLOWER]: Refusing update from remote peer 60871e8ec112431f84c5183cef800239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:19:44.997822 18843 raft_consensus.cc:1261] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8 [term 1 FOLLOWER]: Refusing update from remote peer 60871e8ec112431f84c5183cef800239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:19:44.998147 19044 consensus_queue.cc:1035] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cfa957cb5914007b99df2032731a574" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 37737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:19:44.998286 19041 consensus_queue.cc:1035] T 0ce1ffbf5beb467daed30fa930220b65 P 60871e8ec112431f84c5183cef800239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "286b0cfe82484b7691977229f5d54bc8" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 40789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:19:45.000892 19048 mvcc.cc:205] Tried to move back new op lower bound from 7333035970548277248 to 7333035970408431616. Current Snapshot: MvccSnapshot[applied={T|T < 7333035970548277248}]
I20260924 23:19:45.001004 19049 mvcc.cc:205] Tried to move back new op lower bound from 7333035970548277248 to 7333035970408431616. Current Snapshot: MvccSnapshot[applied={T|T < 7333035970548277248}]
I20260924 23:19:45.003257 18568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:19:45.003355 18568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:19:45.004163 18568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5092e10ae42c492fb8559bda412248ed: Sending DeleteTablet for 3 replicas of tablet 0ce1ffbf5beb467daed30fa930220b65
I20260924 23:19:45.004482 18823 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ce1ffbf5beb467daed30fa930220b65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:44 UTC) from {username='slave'} at 127.0.0.1:41120
I20260924 23:19:45.004556 18954 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ce1ffbf5beb467daed30fa930220b65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:44 UTC) from {username='slave'} at 127.0.0.1:43642
I20260924 23:19:45.004748 19058 tablet_replica.cc:330] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574: stopping tablet replica
I20260924 23:19:45.004812 18692 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ce1ffbf5beb467daed30fa930220b65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:44 UTC) from {username='slave'} at 127.0.0.1:58494
I20260924 23:19:45.004848 19058 raft_consensus.cc:2229] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:19:45.004889 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 18630
I20260924 23:19:45.004946 19058 raft_consensus.cc:2258] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:19:45.007031 19058 ts_tablet_manager.cc:1905] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:19:45.007494 19057 tablet_replica.cc:330] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8: stopping tablet replica
I20260924 23:19:45.007602 19057 raft_consensus.cc:2229] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:19:45.007712 19057 raft_consensus.cc:2258] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:19:45.008203 19057 ts_tablet_manager.cc:1905] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:19:45.008878 19058 ts_tablet_manager.cc:1918] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:19:45.008946 19058 log.cc:1196] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/0ce1ffbf5beb467daed30fa930220b65
I20260924 23:19:45.009176 19058 ts_tablet_manager.cc:1939] T 0ce1ffbf5beb467daed30fa930220b65 P 6cfa957cb5914007b99df2032731a574: Deleting consensus metadata
I20260924 23:19:45.009507 19057 ts_tablet_manager.cc:1918] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:19:45.009579 19057 log.cc:1196] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/0ce1ffbf5beb467daed30fa930220b65
I20260924 23:19:45.009598 18555 catalog_manager.cc:4841] TS 6cfa957cb5914007b99df2032731a574 (127.1.4.195:37737): tablet 0ce1ffbf5beb467daed30fa930220b65 (table test-table [id=e535570bf77a4268b2173e73e07426b9]) successfully deleted
I20260924 23:19:45.009788 19057 ts_tablet_manager.cc:1939] T 0ce1ffbf5beb467daed30fa930220b65 P 286b0cfe82484b7691977229f5d54bc8: Deleting consensus metadata
I20260924 23:19:45.010514 18554 catalog_manager.cc:4841] TS 286b0cfe82484b7691977229f5d54bc8 (127.1.4.194:40789): tablet 0ce1ffbf5beb467daed30fa930220b65 (table test-table [id=e535570bf77a4268b2173e73e07426b9]) successfully deleted
W20260924 23:19:45.010651 18555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.4.193:33125 (error 108)
I20260924 23:19:45.010727 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 18761
I20260924 23:19:45.011543 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:45.011429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:33125 (local address 127.0.0.1:58518)
0924 23:19:45.011468 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:45.011509 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:33125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 23:19:45.011658 18555 catalog_manager.cc:4584] TS 60871e8ec112431f84c5183cef800239 (127.1.4.193:33125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ce1ffbf5beb467daed30fa930220b65: Network error: Client connection negotiation failed: client connection to 127.1.4.193:33125: connect: Connection refused (error 111)
I20260924 23:19:45.015441 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 18892
I20260924 23:19:45.019553 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 18539
2026-09-24T23:19:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (787 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:19:45 dist-test-slave-s1hw krb5kdc[19063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:19:45 dist-test-slave-s1hw krb5kdc[19063](info): set up 2 sockets
Sep 24 23:19:45 dist-test-slave-s1hw krb5kdc[19063](info): commencing operation
krb5kdc: starting...
W20260924 23:19:45.982899 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:19:46 dist-test-slave-s1hw krb5kdc[19063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:19:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:19:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:19:46.104202 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:43937
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:43937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:46.185132 19076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:46.185361 19076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:46.185384 19076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:46.186729 19076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:19:46.186782 19076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:46.186796 19076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:46.186808 19076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:19:46.186820 19076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:19:46.188416 19076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:43937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:43937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:46.188661 19076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:46.188930 19076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:46.193838 19081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:46.193877 19076 server_base.cc:1034] running on GCE node
W20260924 23:19:46.193845 19084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:46.193845 19082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:46.194262 19076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:46.194525 19076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:46.195655 19076 hybrid_clock.cc:648] HybridClock initialized: now 1790291986195670 us; error 57 us; skew 500 ppm
Sep 24 23:19:46 dist-test-slave-s1hw krb5kdc[19063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:46.198295 19076 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:19:46.198690 19076 webserver.cc:460] Webserver started at http://127.1.4.254:45623/ using document root <none> and password file <none>
I20260924 23:19:46.198891 19076 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:46.198930 19076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:46.199019 19076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:46.200052 19076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "9c10b18a01f94957ac5a3baa402af77f"
format_stamp: "Formatted at 2026-09-24 23:19:46 on dist-test-slave-s1hw"
server_key: "0f3c352a5e8892c2c6895f1a2227555a"
server_key_iv: "7a4eccc62e938639201822c1ce2b1d16"
server_key_version: "encryptionkey@0"
I20260924 23:19:46.200342 19076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "9c10b18a01f94957ac5a3baa402af77f"
format_stamp: "Formatted at 2026-09-24 23:19:46 on dist-test-slave-s1hw"
server_key: "0f3c352a5e8892c2c6895f1a2227555a"
server_key_iv: "7a4eccc62e938639201822c1ce2b1d16"
server_key_version: "encryptionkey@0"
I20260924 23:19:46.201591 19076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:19:46.202292 19091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:46.202482 19076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:46.202656 19076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "9c10b18a01f94957ac5a3baa402af77f"
format_stamp: "Formatted at 2026-09-24 23:19:46 on dist-test-slave-s1hw"
server_key: "0f3c352a5e8892c2c6895f1a2227555a"
server_key_iv: "7a4eccc62e938639201822c1ce2b1d16"
server_key_version: "encryptionkey@0"
I20260924 23:19:46.202733 19076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:46.233443 19076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:46.235930 19076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:46.236086 19076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:46.240272 19076 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:43937
I20260924 23:19:46.240324 19143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:43937 every 8 connection(s)
I20260924 23:19:46.240701 19076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:19:46.241493 19144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:46.243805 19144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f: Bootstrap starting.
I20260924 23:19:46.244354 19144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:46.244608 19144 log.cc:824] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:46.245318 19144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f: No bootstrap required, opened a new log
I20260924 23:19:46.246904 19144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c10b18a01f94957ac5a3baa402af77f" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 43937 } }
I20260924 23:19:46.247038 19144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:46.247058 19144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c10b18a01f94957ac5a3baa402af77f, State: Initialized, Role: FOLLOWER
I20260924 23:19:46.247200 19144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c10b18a01f94957ac5a3baa402af77f" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 43937 } }
I20260924 23:19:46.247269 19144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:19:46.247291 19144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:19:46.247336 19144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:46.247871 19144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c10b18a01f94957ac5a3baa402af77f" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 43937 } }
I20260924 23:19:46.247956 19144 leader_election.cc:302] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c10b18a01f94957ac5a3baa402af77f; no voters:
I20260924 23:19:46.248071 19144 leader_election.cc:288] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:19:46.248106 19147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:46.248232 19147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [term 1 LEADER]: Becoming Leader. State: Replica: 9c10b18a01f94957ac5a3baa402af77f, State: Running, Role: LEADER
I20260924 23:19:46.248310 19144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:19:46.248323 19147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c10b18a01f94957ac5a3baa402af77f" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 43937 } }
I20260924 23:19:46.248665 19149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c10b18a01f94957ac5a3baa402af77f. Latest consensus state: current_term: 1 leader_uuid: "9c10b18a01f94957ac5a3baa402af77f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c10b18a01f94957ac5a3baa402af77f" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 43937 } } }
I20260924 23:19:46.248740 19149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:46.248790 19148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c10b18a01f94957ac5a3baa402af77f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c10b18a01f94957ac5a3baa402af77f" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 43937 } } }
I20260924 23:19:46.248849 19148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:46.248947 19154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:19:46.249274 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 19076
I20260924 23:19:46.249389 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:19:46.249426 19154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:19:46.249574 1043 external_mini_cluster.cc:1428] Setting key 25161f0074a2b8e8eca37530080d7f70
I20260924 23:19:46.251070 19154 catalog_manager.cc:1347] Generated new cluster ID: bb641543f0b24018959cbecc9b459893
I20260924 23:19:46.251122 19154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:19:46 dist-test-slave-s1hw krb5kdc[19063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:19:46.259089 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.250044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.1.4.254:43937)
0924 23:19:46.250444 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:46.250453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:46.250485 (+ 32us) server_negotiation.cc:407] Connection header received
0924 23:19:46.250595 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:46.250603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:46.250668 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:46.250808 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:46.251546 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.252495 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.253361 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.253784 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.253874 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.256118 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:46.256122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:46.256123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:46.256135 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:46.257649 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:46.257963 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:46.257966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:46.257966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:46.258057 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:46.258231 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:46.258232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:46.258232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:46.258494 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:46.258645 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:46.258772 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 23:19:46.258911 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":62,"threads_started":1}
I20260924 23:19:46.289199 19154 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:19:46.289772 19154 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:19:46.293661 19154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f: Generated new TSK 0
I20260924 23:19:46.293857 19154 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:19:46.327126 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:43937
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:46.426211 19170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:46.426429 19170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:46.426451 19170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:46.427626 19170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:46.427671 19170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:46.427696 19170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:19:46.429118 19170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:43937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:46.429312 19170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:46.429492 19170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:46.434716 19176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:46.434719 19178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:46.434888 19170 server_base.cc:1034] running on GCE node
W20260924 23:19:46.434804 19175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:46.435196 19170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:46.435465 19170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:46.436616 19170 hybrid_clock.cc:648] HybridClock initialized: now 1790291986436604 us; error 29 us; skew 500 ppm
Sep 24 23:19:46 dist-test-slave-s1hw krb5kdc[19063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:46.439035 19170 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:19:46.439556 19170 webserver.cc:460] Webserver started at http://127.1.4.193:34961/ using document root <none> and password file <none>
I20260924 23:19:46.439771 19170 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:46.439818 19170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:46.439941 19170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:46.441009 19170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "85613c6f273e41149351d11498d5a4ab"
format_stamp: "Formatted at 2026-09-24 23:19:46 on dist-test-slave-s1hw"
server_key: "d1981ef5d19b357cc888dd4d50a1b40c"
server_key_iv: "df2aeca346a69be8f814f457f3880858"
server_key_version: "encryptionkey@0"
I20260924 23:19:46.441339 19170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "85613c6f273e41149351d11498d5a4ab"
format_stamp: "Formatted at 2026-09-24 23:19:46 on dist-test-slave-s1hw"
server_key: "d1981ef5d19b357cc888dd4d50a1b40c"
server_key_iv: "df2aeca346a69be8f814f457f3880858"
server_key_version: "encryptionkey@0"
I20260924 23:19:46.442556 19170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:19:46.443251 19185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:46.443457 19170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:46.443578 19170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "85613c6f273e41149351d11498d5a4ab"
format_stamp: "Formatted at 2026-09-24 23:19:46 on dist-test-slave-s1hw"
server_key: "d1981ef5d19b357cc888dd4d50a1b40c"
server_key_iv: "df2aeca346a69be8f814f457f3880858"
server_key_version: "encryptionkey@0"
I20260924 23:19:46.443650 19170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:46.452544 19170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:46.454296 19170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:46.454401 19170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:46.454567 19170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:46.454873 19170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:46.454912 19170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:46.454939 19170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:46.454957 19170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:46.460484 19170 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:40979
I20260924 23:19:46.460541 19298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:40979 every 8 connection(s)
I20260924 23:19:46.460834 19170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:19:46.462318 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 19170
I20260924 23:19:46.462440 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:19:46.462626 1043 external_mini_cluster.cc:1428] Setting key fbb234dffbb11f56e2a2f7677a8b9e26
Sep 24 23:19:46 dist-test-slave-s1hw krb5kdc[19063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
I20260924 23:19:46.470461 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.461503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:59541 (local address 127.1.4.254:43937)
0924 23:19:46.461566 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:46.461568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:46.461761 (+ 193us) server_negotiation.cc:407] Connection header received
0924 23:19:46.462259 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:46.462260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:46.462275 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:46.462356 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:46.463304 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.463861 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.465138 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.465348 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.465412 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.467528 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:46.467529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:46.467530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:46.467538 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:46.469120 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:46.469536 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:46.469538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:46.469538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:46.469592 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:46.469815 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:46.469816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:46.469816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:46.469927 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:46.469984 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:46.470263 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 23:19:46.470366 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 23:19:46.470623 19301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.461475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:43937 (local address 127.1.4.193:59541)
0924 23:19:46.461633 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:46.461674 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:46.462143 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:46.462411 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:46.462414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:46.463010 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:46.463206 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:46.463210 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.463949 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.463950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:46.465012 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:46.465015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.465438 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.465439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:46.465551 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.466064 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:19:46.466073 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:46.467438 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:46.469227 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:46.469235 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:46.469238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:46.469461 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:46.469653 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:46.469653 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:46.469654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:46.469756 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:46.469984 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:46.469991 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:46.470131 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 23:19:46.470228 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 23:19:46.470407 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260924 23:19:46.471072 19299 heartbeater.cc:347] Connected to a master server at 127.1.4.254:43937
I20260924 23:19:46.471201 19299 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:46.471406 19299 heartbeater.cc:511] Master 127.1.4.254:43937 requested a full tablet report, sending...
I20260924 23:19:46.471884 19108 ts_manager.cc:194] Registered new tserver with Master: 85613c6f273e41149351d11498d5a4ab (127.1.4.193:40979)
I20260924 23:19:46.472545 19108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:59541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:19:46.490176 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:43937
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:46.569608 19304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:46.569832 19304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:46.569864 19304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:46.571074 19304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:46.571122 19304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:46.571157 19304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:19:46.572613 19304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:43937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:46.572850 19304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:46.573066 19304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:46.577979 19312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:46.578022 19310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:46.578083 19309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:46.578150 19304 server_base.cc:1034] running on GCE node
I20260924 23:19:46.578368 19304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:46.578594 19304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:46.579735 19304 hybrid_clock.cc:648] HybridClock initialized: now 1790291986579718 us; error 37 us; skew 500 ppm
Sep 24 23:19:46 dist-test-slave-s1hw krb5kdc[19063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:46.582187 19304 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:19:46.582623 19304 webserver.cc:460] Webserver started at http://127.1.4.194:42101/ using document root <none> and password file <none>
I20260924 23:19:46.582834 19304 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:46.582880 19304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:46.583000 19304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:46.584059 19304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "de57b77602954506acf97468d83beee5"
format_stamp: "Formatted at 2026-09-24 23:19:46 on dist-test-slave-s1hw"
server_key: "97e04d82f621fbfe64abcb1a6fd46393"
server_key_iv: "06402553dcc4fca2ce435d8c688aa86b"
server_key_version: "encryptionkey@0"
I20260924 23:19:46.584408 19304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "de57b77602954506acf97468d83beee5"
format_stamp: "Formatted at 2026-09-24 23:19:46 on dist-test-slave-s1hw"
server_key: "97e04d82f621fbfe64abcb1a6fd46393"
server_key_iv: "06402553dcc4fca2ce435d8c688aa86b"
server_key_version: "encryptionkey@0"
I20260924 23:19:46.585600 19304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:19:46.586246 19319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:46.586403 19304 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:46.586476 19304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "de57b77602954506acf97468d83beee5"
format_stamp: "Formatted at 2026-09-24 23:19:46 on dist-test-slave-s1hw"
server_key: "97e04d82f621fbfe64abcb1a6fd46393"
server_key_iv: "06402553dcc4fca2ce435d8c688aa86b"
server_key_version: "encryptionkey@0"
I20260924 23:19:46.586539 19304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:46.595160 19304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:46.597448 19304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:46.597595 19304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:46.597884 19304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:46.598327 19304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:46.598373 19304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:46.598402 19304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:46.598426 19304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:46.604292 19304 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:43097
I20260924 23:19:46.604353 19432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:43097 every 8 connection(s)
I20260924 23:19:46.604633 19304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:19:46 dist-test-slave-s1hw krb5kdc[19063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:19:46.613487 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.605368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:46891 (local address 127.1.4.254:43937)
0924 23:19:46.605433 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:46.605435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:46.605621 (+ 186us) server_negotiation.cc:407] Connection header received
0924 23:19:46.606123 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:46.606125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:46.606137 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:46.606184 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:46.607178 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.607805 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.608780 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.608986 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.609042 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.611054 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:46.611055 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:46.611056 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:46.611063 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:46.612219 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:46.612624 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:46.612625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:46.612626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:46.612678 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:46.612896 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:46.612897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:46.612897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:46.613002 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:46.613057 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:46.613315 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 23:19:46.613396 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 23:19:46.613646 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.605356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:43937 (local address 127.1.4.194:46891)
0924 23:19:46.605502 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:46.605540 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:46.606023 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:46.606284 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:46.606288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:46.606885 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:46.607081 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:46.607086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.607889 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.607891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:46.608690 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:46.608693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.609054 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.609055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:46.609175 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.609599 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:19:46.609606 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:46.610966 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:46.612316 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:46.612322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:46.612324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:46.612557 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:46.612741 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:46.612742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:46.612742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:46.612840 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:46.613058 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:46.613064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:46.613196 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 23:19:46.613286 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 23:19:46.613438 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260924 23:19:46.614053 19433 heartbeater.cc:347] Connected to a master server at 127.1.4.254:43937
I20260924 23:19:46.614144 19433 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:46.614318 19433 heartbeater.cc:511] Master 127.1.4.254:43937 requested a full tablet report, sending...
I20260924 23:19:46.614669 19108 ts_manager.cc:194] Registered new tserver with Master: de57b77602954506acf97468d83beee5 (127.1.4.194:43097)
I20260924 23:19:46.614797 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 19304
I20260924 23:19:46.614900 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:19:46.615075 1043 external_mini_cluster.cc:1428] Setting key bdca67a8dc0bd1d44e81e13045fe49b9
I20260924 23:19:46.615096 19108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:46891
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:19:46.638334 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:43937
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:46.716642 19438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:46.716850 19438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:46.716881 19438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:46.717996 19438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:46.718036 19438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:46.718070 19438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:19:46.719466 19438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:43937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:46.719696 19438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:46.719905 19438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:46.724582 19443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:46.724624 19444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:46.724614 19446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:46.724839 19438 server_base.cc:1034] running on GCE node
I20260924 23:19:46.724999 19438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:46.725184 19438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:46.726325 19438 hybrid_clock.cc:648] HybridClock initialized: now 1790291986726315 us; error 26 us; skew 500 ppm
Sep 24 23:19:46 dist-test-slave-s1hw krb5kdc[19063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:46.728662 19438 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:19:46.729029 19438 webserver.cc:460] Webserver started at http://127.1.4.195:35745/ using document root <none> and password file <none>
I20260924 23:19:46.729214 19438 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:46.729254 19438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:46.729341 19438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:46.730307 19438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "3e72f1b4f57e410cae5b1fe6fe4f19f3"
format_stamp: "Formatted at 2026-09-24 23:19:46 on dist-test-slave-s1hw"
server_key: "18621da5efd31324a564d30ebe4a01e8"
server_key_iv: "9388c10ab8ebfd5c20a43c32501c2247"
server_key_version: "encryptionkey@0"
I20260924 23:19:46.730581 19438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "3e72f1b4f57e410cae5b1fe6fe4f19f3"
format_stamp: "Formatted at 2026-09-24 23:19:46 on dist-test-slave-s1hw"
server_key: "18621da5efd31324a564d30ebe4a01e8"
server_key_iv: "9388c10ab8ebfd5c20a43c32501c2247"
server_key_version: "encryptionkey@0"
I20260924 23:19:46.731815 19438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:19:46.732470 19453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:46.732604 19438 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:46.732671 19438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "3e72f1b4f57e410cae5b1fe6fe4f19f3"
format_stamp: "Formatted at 2026-09-24 23:19:46 on dist-test-slave-s1hw"
server_key: "18621da5efd31324a564d30ebe4a01e8"
server_key_iv: "9388c10ab8ebfd5c20a43c32501c2247"
server_key_version: "encryptionkey@0"
I20260924 23:19:46.732723 19438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:46.740337 19438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:46.742264 19438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:46.742364 19438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:46.742530 19438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:46.742758 19438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:46.742786 19438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:46.742805 19438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:46.742821 19438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:46.748258 19438 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:35505
I20260924 23:19:46.748342 19566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:35505 every 8 connection(s)
I20260924 23:19:46.748610 19438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:19:46.752410 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 19438
I20260924 23:19:46.752522 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:19:46.752712 1043 external_mini_cluster.cc:1428] Setting key 3248378fc5f9390e8f4ef92494602bc2
Sep 24 23:19:46 dist-test-slave-s1hw krb5kdc[19063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:19:46.757277 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.749392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:56573 (local address 127.1.4.254:43937)
0924 23:19:46.749452 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:46.749454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:46.749642 (+ 188us) server_negotiation.cc:407] Connection header received
0924 23:19:46.750144 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:46.750145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:46.750156 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:46.750221 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:46.751100 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.751662 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.752701 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.752944 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.753009 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.755122 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:46.755123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:46.755124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:46.755130 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:46.756252 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:46.756585 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:46.756586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:46.756586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:46.756629 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:46.756817 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:46.756817 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:46.756818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:46.756909 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:46.756953 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:46.757148 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 23:19:46.757216 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 23:19:46.757488 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.749373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:43937 (local address 127.1.4.195:56573)
0924 23:19:46.749537 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:46.749574 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:46.750060 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:46.750278 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:46.750281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:46.750843 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:46.751034 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:46.751038 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.751766 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.751767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:46.752606 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:46.752609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.753028 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.753030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:46.753180 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.753681 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:19:46.753691 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:46.755040 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:46.756317 (+ 1277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:46.756322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:46.756323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:46.756527 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:46.756674 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:46.756675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:46.756675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:46.756771 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:46.756952 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:46.756955 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:46.757063 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 23:19:46.757137 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 23:19:46.757273 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260924 23:19:46.757885 19567 heartbeater.cc:347] Connected to a master server at 127.1.4.254:43937
I20260924 23:19:46.757964 19567 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:46.758126 19567 heartbeater.cc:511] Master 127.1.4.254:43937 requested a full tablet report, sending...
I20260924 23:19:46.758455 19108 ts_manager.cc:194] Registered new tserver with Master: 3e72f1b4f57e410cae5b1fe6fe4f19f3 (127.1.4.195:35505)
I20260924 23:19:46.758807 19108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:56573
I20260924 23:19:46.764648 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:19:46.771960 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.766026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.1.4.254:43937)
0924 23:19:46.766098 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:46.766100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:46.766210 (+ 110us) server_negotiation.cc:407] Connection header received
0924 23:19:46.766249 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:46.766250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:46.766260 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:46.766305 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:46.766811 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.767293 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.768055 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.768258 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.768312 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.769544 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:46.769546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:46.769546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:46.769553 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:46.770769 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:46.771148 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:46.771149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:46.771149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:46.771211 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:46.771418 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:46.771419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:46.771419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:46.771563 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:46.771659 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:46.771806 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 23:19:46.771867 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 23:19:46.773406 19108 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:19:46.775050 19108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:19:46.781176 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.777298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.1.4.195:35505)
0924 23:19:46.777897 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:46.777901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:46.777910 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:19:46.777945 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:46.777947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:46.777961 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:46.778156 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:46.778239 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.779226 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.780298 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.780784 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.780850 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.780914 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:46.781053 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 23:19:46.781094 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":84,"threads_started":1}
I20260924 23:19:46.781165 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.777056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:35505 (local address 127.0.0.1:49378)
0924 23:19:46.777253 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:46.777275 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:46.777327 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:46.778052 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:46.778053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:46.778055 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:46.778093 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:46.778095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.779312 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.779313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:46.780217 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:46.780219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.780869 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.780871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:46.780947 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.780961 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 23:19:46.781026 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 23:19:46.781081 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"spinlock_wait_cycles":768,"thread_start_us":60,"threads_started":1}
I20260924 23:19:46.781335 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.777489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40008 (local address 127.1.4.194:43097)
0924 23:19:46.777625 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:46.777629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:46.777637 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:19:46.777667 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:46.777669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:46.777680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:46.777748 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:46.777950 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.778708 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.780123 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.780833 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.780898 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.780952 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:46.781217 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 23:19:46.781256 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20260924 23:19:46.781502 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.777319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50122 (local address 127.1.4.193:40979)
0924 23:19:46.777644 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:46.777648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:46.777656 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:19:46.777686 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:46.777688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:46.777702 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:46.777777 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:46.777947 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.779044 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.780327 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.781001 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.781058 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.781105 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:46.781389 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 23:19:46.781430 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":74,"threads_started":1}
I20260924 23:19:46.781644 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.777061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:43097 (local address 127.0.0.1:40008)
0924 23:19:46.777401 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:46.777408 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:46.777440 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:46.777778 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:46.777781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:46.777783 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:46.777871 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:46.777873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.778818 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.778820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:46.780048 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:46.780051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.780988 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.780989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:46.781058 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.781062 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:19:46.781104 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 23:19:46.781147 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":3,"thread_start_us":67,"threads_started":1}
I20260924 23:19:46.781322 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.777055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:40979 (local address 127.0.0.1:50122)
0924 23:19:46.777221 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:46.777259 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:46.777327 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:46.777792 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:46.777794 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:46.777796 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:46.777868 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:46.777870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.779147 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.779148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:46.780266 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:46.780269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.781132 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.781133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:46.781181 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.781185 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:19:46.781229 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:19:46.781259 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":60,"thread_start_us":61,"threads_started":1}
I20260924 23:19:46.782364 19233 tablet_service.cc:1467] Processing CreateTablet for tablet d63db5b663504e1f99a387f823448280 (DEFAULT_TABLE table=test-table [id=ff77edd81f9c46fa9673044ad4c4f628]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:46.782372 19501 tablet_service.cc:1467] Processing CreateTablet for tablet d63db5b663504e1f99a387f823448280 (DEFAULT_TABLE table=test-table [id=ff77edd81f9c46fa9673044ad4c4f628]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:46.782691 19233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63db5b663504e1f99a387f823448280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:46.782697 19501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63db5b663504e1f99a387f823448280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:46.785135 19582 tablet_bootstrap.cc:492] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab: Bootstrap starting.
I20260924 23:19:46.785585 19367 tablet_service.cc:1467] Processing CreateTablet for tablet d63db5b663504e1f99a387f823448280 (DEFAULT_TABLE table=test-table [id=ff77edd81f9c46fa9673044ad4c4f628]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:46.785863 19582 tablet_bootstrap.cc:654] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:46.785899 19367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d63db5b663504e1f99a387f823448280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:46.786394 19582 log.cc:824] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:46.786747 19581 tablet_bootstrap.cc:492] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3: Bootstrap starting.
I20260924 23:19:46.787128 19582 tablet_bootstrap.cc:492] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab: No bootstrap required, opened a new log
I20260924 23:19:46.787190 19582 ts_tablet_manager.cc:1397] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:19:46.787400 19581 tablet_bootstrap.cc:654] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:46.787897 19581 log.cc:824] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:46.788698 19581 tablet_bootstrap.cc:492] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3: No bootstrap required, opened a new log
I20260924 23:19:46.788780 19581 ts_tablet_manager.cc:1397] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:19:46.788967 19582 raft_consensus.cc:348] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85613c6f273e41149351d11498d5a4ab" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 40979 } } peers { permanent_uuid: "de57b77602954506acf97468d83beee5" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 43097 } } peers { permanent_uuid: "3e72f1b4f57e410cae5b1fe6fe4f19f3" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 35505 } }
I20260924 23:19:46.789151 19582 raft_consensus.cc:374] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:46.789192 19582 raft_consensus.cc:729] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85613c6f273e41149351d11498d5a4ab, State: Initialized, Role: FOLLOWER
I20260924 23:19:46.789321 19582 consensus_queue.cc:260] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85613c6f273e41149351d11498d5a4ab" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 40979 } } peers { permanent_uuid: "de57b77602954506acf97468d83beee5" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 43097 } } peers { permanent_uuid: "3e72f1b4f57e410cae5b1fe6fe4f19f3" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 35505 } }
I20260924 23:19:46.789387 19585 tablet_bootstrap.cc:492] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5: Bootstrap starting.
I20260924 23:19:46.789604 19582 ts_tablet_manager.cc:1428] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:19:46.789758 19299 heartbeater.cc:503] Master 127.1.4.254:43937 was elected leader, sending a full tablet report...
I20260924 23:19:46.790015 19585 tablet_bootstrap.cc:654] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:46.790609 19585 log.cc:824] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:46.790571 19581 raft_consensus.cc:348] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85613c6f273e41149351d11498d5a4ab" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 40979 } } peers { permanent_uuid: "de57b77602954506acf97468d83beee5" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 43097 } } peers { permanent_uuid: "3e72f1b4f57e410cae5b1fe6fe4f19f3" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 35505 } }
I20260924 23:19:46.790733 19581 raft_consensus.cc:374] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:46.790774 19581 raft_consensus.cc:729] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e72f1b4f57e410cae5b1fe6fe4f19f3, State: Initialized, Role: FOLLOWER
I20260924 23:19:46.790902 19581 consensus_queue.cc:260] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85613c6f273e41149351d11498d5a4ab" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 40979 } } peers { permanent_uuid: "de57b77602954506acf97468d83beee5" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 43097 } } peers { permanent_uuid: "3e72f1b4f57e410cae5b1fe6fe4f19f3" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 35505 } }
I20260924 23:19:46.791177 19581 ts_tablet_manager.cc:1428] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:19:46.791301 19567 heartbeater.cc:503] Master 127.1.4.254:43937 was elected leader, sending a full tablet report...
I20260924 23:19:46.791435 19585 tablet_bootstrap.cc:492] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5: No bootstrap required, opened a new log
I20260924 23:19:46.791510 19585 ts_tablet_manager.cc:1397] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:19:46.792923 19585 raft_consensus.cc:348] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85613c6f273e41149351d11498d5a4ab" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 40979 } } peers { permanent_uuid: "de57b77602954506acf97468d83beee5" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 43097 } } peers { permanent_uuid: "3e72f1b4f57e410cae5b1fe6fe4f19f3" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 35505 } }
I20260924 23:19:46.793063 19585 raft_consensus.cc:374] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:46.793107 19585 raft_consensus.cc:729] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de57b77602954506acf97468d83beee5, State: Initialized, Role: FOLLOWER
I20260924 23:19:46.793216 19585 consensus_queue.cc:260] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85613c6f273e41149351d11498d5a4ab" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 40979 } } peers { permanent_uuid: "de57b77602954506acf97468d83beee5" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 43097 } } peers { permanent_uuid: "3e72f1b4f57e410cae5b1fe6fe4f19f3" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 35505 } }
I20260924 23:19:46.793442 19585 ts_tablet_manager.cc:1428] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:19:46.793489 19433 heartbeater.cc:503] Master 127.1.4.254:43937 was elected leader, sending a full tablet report...
W20260924 23:19:46.855304 19434 tablet.cc:2369] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:19:46.888880 19589 raft_consensus.cc:482] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:19:46.889020 19589 raft_consensus.cc:504] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85613c6f273e41149351d11498d5a4ab" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 40979 } } peers { permanent_uuid: "de57b77602954506acf97468d83beee5" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 43097 } } peers { permanent_uuid: "3e72f1b4f57e410cae5b1fe6fe4f19f3" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 35505 } }
I20260924 23:19:46.889415 19589 leader_election.cc:288] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85613c6f273e41149351d11498d5a4ab (127.1.4.193:40979), 3e72f1b4f57e410cae5b1fe6fe4f19f3 (127.1.4.195:35505)
I20260924 23:19:46.893585 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.889678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:35551 (local address 127.1.4.195:35505)
0924 23:19:46.889728 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:46.889730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:46.889770 (+ 40us) server_negotiation.cc:407] Connection header received
0924 23:19:46.889843 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:46.889844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:46.889857 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:46.889920 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:46.890080 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.890845 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.892480 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.893111 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.893194 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.893207 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:46.893479 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 23:19:46.893527 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:19:46.893584 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.889678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:37047 (local address 127.1.4.193:40979)
0924 23:19:46.889785 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:46.889787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:46.889878 (+ 91us) server_negotiation.cc:407] Connection header received
0924 23:19:46.889917 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:46.889918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:46.889928 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:46.889973 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:46.890104 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.890849 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.892480 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.893160 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.893216 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.893226 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:46.893471 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 23:19:46.893518 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 23:19:46.893584 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.889602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:40979 (local address 127.1.4.194:37047)
0924 23:19:46.889809 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:46.889818 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:46.889869 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:46.889990 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:46.889991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:46.889992 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:46.890042 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:46.890043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.890980 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.890982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:46.892346 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:46.892351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.893307 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.893308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:46.893368 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.893373 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:19:46.893415 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 23:19:46.893458 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":24,"thread_start_us":117,"threads_started":1}
I20260924 23:19:46.893589 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.889602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:35505 (local address 127.1.4.194:35551)
0924 23:19:46.889658 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:46.889716 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:46.889755 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:46.889937 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:46.889938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:46.889939 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:46.890008 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:46.890010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.890989 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.890989 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:46.892346 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:46.892351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.893242 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.893243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:46.893370 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.893374 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:19:46.893454 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 23:19:46.893505 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":105}
I20260924 23:19:46.893942 19521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63db5b663504e1f99a387f823448280" candidate_uuid: "de57b77602954506acf97468d83beee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e72f1b4f57e410cae5b1fe6fe4f19f3" is_pre_election: true
I20260924 23:19:46.894104 19521 raft_consensus.cc:2454] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de57b77602954506acf97468d83beee5 in term 0.
I20260924 23:19:46.894121 19253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63db5b663504e1f99a387f823448280" candidate_uuid: "de57b77602954506acf97468d83beee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85613c6f273e41149351d11498d5a4ab" is_pre_election: true
I20260924 23:19:46.894284 19253 raft_consensus.cc:2454] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de57b77602954506acf97468d83beee5 in term 0.
I20260924 23:19:46.894333 19323 leader_election.cc:302] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e72f1b4f57e410cae5b1fe6fe4f19f3, de57b77602954506acf97468d83beee5; no voters:
I20260924 23:19:46.894460 19589 raft_consensus.cc:2781] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:19:46.894526 19589 raft_consensus.cc:482] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:19:46.894551 19589 raft_consensus.cc:3037] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:46.895269 19589 raft_consensus.cc:504] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85613c6f273e41149351d11498d5a4ab" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 40979 } } peers { permanent_uuid: "de57b77602954506acf97468d83beee5" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 43097 } } peers { permanent_uuid: "3e72f1b4f57e410cae5b1fe6fe4f19f3" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 35505 } }
I20260924 23:19:46.895421 19589 leader_election.cc:288] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [CANDIDATE]: Term 1 election: Requested vote from peers 85613c6f273e41149351d11498d5a4ab (127.1.4.193:40979), 3e72f1b4f57e410cae5b1fe6fe4f19f3 (127.1.4.195:35505)
I20260924 23:19:46.895589 19521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63db5b663504e1f99a387f823448280" candidate_uuid: "de57b77602954506acf97468d83beee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e72f1b4f57e410cae5b1fe6fe4f19f3"
I20260924 23:19:46.895599 19253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d63db5b663504e1f99a387f823448280" candidate_uuid: "de57b77602954506acf97468d83beee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85613c6f273e41149351d11498d5a4ab"
I20260924 23:19:46.895676 19521 raft_consensus.cc:3037] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:46.895676 19253 raft_consensus.cc:3037] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:46.896559 19521 raft_consensus.cc:2454] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de57b77602954506acf97468d83beee5 in term 1.
I20260924 23:19:46.896559 19253 raft_consensus.cc:2454] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de57b77602954506acf97468d83beee5 in term 1.
I20260924 23:19:46.896726 19322 leader_election.cc:302] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85613c6f273e41149351d11498d5a4ab, de57b77602954506acf97468d83beee5; no voters:
I20260924 23:19:46.896811 19589 raft_consensus.cc:2781] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:46.896977 19589 raft_consensus.cc:686] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [term 1 LEADER]: Becoming Leader. State: Replica: de57b77602954506acf97468d83beee5, State: Running, Role: LEADER
I20260924 23:19:46.897075 19589 consensus_queue.cc:237] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85613c6f273e41149351d11498d5a4ab" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 40979 } } peers { permanent_uuid: "de57b77602954506acf97468d83beee5" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 43097 } } peers { permanent_uuid: "3e72f1b4f57e410cae5b1fe6fe4f19f3" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 35505 } }
I20260924 23:19:46.897969 19108 catalog_manager.cc:5495] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 reported cstate change: term changed from 0 to 1, leader changed from <none> to de57b77602954506acf97468d83beee5 (127.1.4.194). New cstate: current_term: 1 leader_uuid: "de57b77602954506acf97468d83beee5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85613c6f273e41149351d11498d5a4ab" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 40979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de57b77602954506acf97468d83beee5" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 43097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e72f1b4f57e410cae5b1fe6fe4f19f3" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 35505 } health_report { overall_health: UNKNOWN } } }
I20260924 23:19:46.926316 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.924209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40010 (local address 127.1.4.194:43097)
0924 23:19:46.924273 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:46.924275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:46.924283 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:19:46.924318 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:46.924319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:46.924338 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:46.924392 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:19:46.924485 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.924947 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.925643 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:46.925839 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:46.925888 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:46.926023 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:19:46.926106 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:19:46.926175 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:46.926209 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 23:19:46.926216 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 23:19:46.928021 19253 raft_consensus.cc:1261] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab [term 1 FOLLOWER]: Refusing update from remote peer de57b77602954506acf97468d83beee5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:19:46.928021 19521 raft_consensus.cc:1261] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3 [term 1 FOLLOWER]: Refusing update from remote peer de57b77602954506acf97468d83beee5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:19:46.928347 19589 consensus_queue.cc:1035] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85613c6f273e41149351d11498d5a4ab" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 40979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:19:46.928501 19591 consensus_queue.cc:1035] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e72f1b4f57e410cae5b1fe6fe4f19f3" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 35505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:19:46.930564 19594 mvcc.cc:205] Tried to move back new op lower bound from 7333035978454577152 to 7333035978331082752. Current Snapshot: MvccSnapshot[applied={T|T < 7333035978454577152}]
I20260924 23:19:46.930912 19596 mvcc.cc:205] Tried to move back new op lower bound from 7333035978454577152 to 7333035978331082752. Current Snapshot: MvccSnapshot[applied={T|T < 7333035978454577152}]
I20260924 23:19:46.933046 19107 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:19:46.933133 19107 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:19:46.933873 19107 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c10b18a01f94957ac5a3baa402af77f: Sending DeleteTablet for 3 replicas of tablet d63db5b663504e1f99a387f823448280
I20260924 23:19:46.934177 19367 tablet_service.cc:1514] Processing DeleteTablet for tablet d63db5b663504e1f99a387f823448280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:46 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:40008
I20260924 23:19:46.934217 19501 tablet_service.cc:1514] Processing DeleteTablet for tablet d63db5b663504e1f99a387f823448280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:46 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:49378
I20260924 23:19:46.934345 19233 tablet_service.cc:1514] Processing DeleteTablet for tablet d63db5b663504e1f99a387f823448280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:46 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:50122
I20260924 23:19:46.934441 19605 tablet_replica.cc:330] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3: stopping tablet replica
I20260924 23:19:46.934541 19605 raft_consensus.cc:2229] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:19:46.934602 19606 tablet_replica.cc:330] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab: stopping tablet replica
I20260924 23:19:46.934648 19605 raft_consensus.cc:2258] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:19:46.934685 19606 raft_consensus.cc:2229] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:19:46.934763 19606 raft_consensus.cc:2258] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:19:46.935067 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 19170
I20260924 23:19:46.935094 19606 ts_tablet_manager.cc:1905] T d63db5b663504e1f99a387f823448280 P 85613c6f273e41149351d11498d5a4ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:19:46.935086 19605 ts_tablet_manager.cc:1905] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:19:46.935626 19604 tablet_replica.cc:330] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5: stopping tablet replica
I20260924 23:19:46.935708 19604 raft_consensus.cc:2229] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:19:46.935788 19604 raft_consensus.cc:2258] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 23:19:46.939388 19094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.4.193:40979 (error 108)
I20260924 23:19:46.939615 19604 ts_tablet_manager.cc:1905] T d63db5b663504e1f99a387f823448280 P de57b77602954506acf97468d83beee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:19:46.940863 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.940700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:40979 (local address 127.0.0.1:50126)
0924 23:19:46.940775 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:46.940824 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 23:19:46.940865 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 19304
W20260924 23:19:46.941787 19094 catalog_manager.cc:4584] TS 85613c6f273e41149351d11498d5a4ab (127.1.4.193:40979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d63db5b663504e1f99a387f823448280: Network error: Client connection negotiation failed: client connection to 127.1.4.193:40979: connect: Connection refused (error 111)
I20260924 23:19:46.943194 19605 ts_tablet_manager.cc:1918] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:19:46.943264 19605 log.cc:1196] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/d63db5b663504e1f99a387f823448280
I20260924 23:19:46.943506 19605 ts_tablet_manager.cc:1939] T d63db5b663504e1f99a387f823448280 P 3e72f1b4f57e410cae5b1fe6fe4f19f3: Deleting consensus metadata
I20260924 23:19:46.943948 19095 catalog_manager.cc:4841] TS 3e72f1b4f57e410cae5b1fe6fe4f19f3 (127.1.4.195:35505): tablet d63db5b663504e1f99a387f823448280 (table test-table [id=ff77edd81f9c46fa9673044ad4c4f628]) successfully deleted
I20260924 23:19:46.945886 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 19438
I20260924 23:19:46.945909 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:46.945800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:43097 (local address 127.0.0.1:40022)
0924 23:19:46.945856 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:46.945881 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.194:43097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 23:19:46.950554 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 19076
2026-09-24T23:19:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1931 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T23:19:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:19:46Z Disabled control of system clock
I20260924 23:19:46.976315 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:45637
--webserver_interface=127.1.4.254
--webserver_port=0
--builtin_ntp_servers=127.1.4.212:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:45637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 23:19:47.055150 19611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:47.055420 19611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:47.055454 19611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:47.056692 19611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:19:47.056739 19611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:47.056751 19611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:47.056762 19611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:19:47.056772 19611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:19:47.058089 19611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:45637
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:45637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:47.058308 19611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:47.058560 19611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:19:47.063323 19611 server_base.cc:1034] running on GCE node
W20260924 23:19:47.063334 19619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:47.063304 19616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:47.063334 19617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:47.063827 19611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:47.064101 19611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:47.065241 19611 hybrid_clock.cc:648] HybridClock initialized: now 1790291987065225 us; error 29 us; skew 500 ppm
I20260924 23:19:47.066520 19611 webserver.cc:460] Webserver started at http://127.1.4.254:35841/ using document root <none> and password file <none>
I20260924 23:19:47.066762 19611 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:47.066814 19611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:47.066942 19611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:47.068058 19611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "b2d9f06ae515497eb54f762521a946af"
format_stamp: "Formatted at 2026-09-24 23:19:47 on dist-test-slave-s1hw"
server_key: "8a56134dc4d257882da0ec985da320fa"
server_key_iv: "2b41ee304355151a15f7e3d13a1d28af"
server_key_version: "encryptionkey@0"
I20260924 23:19:47.068400 19611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "b2d9f06ae515497eb54f762521a946af"
format_stamp: "Formatted at 2026-09-24 23:19:47 on dist-test-slave-s1hw"
server_key: "8a56134dc4d257882da0ec985da320fa"
server_key_iv: "2b41ee304355151a15f7e3d13a1d28af"
server_key_version: "encryptionkey@0"
I20260924 23:19:47.069742 19611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:19:47.070582 19625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:47.070737 19611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:47.070824 19611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "b2d9f06ae515497eb54f762521a946af"
format_stamp: "Formatted at 2026-09-24 23:19:47 on dist-test-slave-s1hw"
server_key: "8a56134dc4d257882da0ec985da320fa"
server_key_iv: "2b41ee304355151a15f7e3d13a1d28af"
server_key_version: "encryptionkey@0"
I20260924 23:19:47.070888 19611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:47.082993 19611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:47.083303 19611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:47.083462 19611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:47.087069 19611 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:45637
I20260924 23:19:47.087112 19677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:45637 every 8 connection(s)
I20260924 23:19:47.087509 19611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:19:47.088452 19678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:47.090215 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 19611
I20260924 23:19:47.090353 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:19:47.090547 1043 external_mini_cluster.cc:1428] Setting key a07c3967eef87da2078ac6b277890ad0
I20260924 23:19:47.091029 19678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af: Bootstrap starting.
I20260924 23:19:47.091694 19678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:47.091982 19678 log.cc:824] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:47.092664 19678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af: No bootstrap required, opened a new log
I20260924 23:19:47.094739 19678 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d9f06ae515497eb54f762521a946af" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45637 } }
I20260924 23:19:47.094878 19678 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:47.094902 19678 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d9f06ae515497eb54f762521a946af, State: Initialized, Role: FOLLOWER
I20260924 23:19:47.095059 19678 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d9f06ae515497eb54f762521a946af" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45637 } }
I20260924 23:19:47.095150 19678 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:19:47.095180 19678 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:19:47.095224 19678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:47.095886 19678 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d9f06ae515497eb54f762521a946af" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45637 } }
I20260924 23:19:47.095978 19678 leader_election.cc:302] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2d9f06ae515497eb54f762521a946af; no voters:
I20260924 23:19:47.096158 19678 leader_election.cc:288] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:19:47.096215 19683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:47.096367 19683 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [term 1 LEADER]: Becoming Leader. State: Replica: b2d9f06ae515497eb54f762521a946af, State: Running, Role: LEADER
I20260924 23:19:47.096446 19678 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:19:47.096453 19683 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d9f06ae515497eb54f762521a946af" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45637 } }
I20260924 23:19:47.096527 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.091152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.1.4.254:45637)
0924 23:19:47.091347 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:47.091354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:47.091408 (+ 54us) server_negotiation.cc:407] Connection header received
0924 23:19:47.091958 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:47.091966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:47.092035 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:47.092190 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:47.092410 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.093335 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.094447 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.094860 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.094982 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.095720 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:47.095723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:47.095724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:47.095890 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:47.095923 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:47.095933 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:47.096007 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:47.096113 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 23:19:47.096310 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260924 23:19:47.096778 19683 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2d9f06ae515497eb54f762521a946af. Latest consensus state: current_term: 1 leader_uuid: "b2d9f06ae515497eb54f762521a946af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d9f06ae515497eb54f762521a946af" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45637 } } }
I20260924 23:19:47.096789 19684 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2d9f06ae515497eb54f762521a946af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d9f06ae515497eb54f762521a946af" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45637 } } }
I20260924 23:19:47.096858 19683 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:47.096879 19684 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af [sys.catalog]: This master's current role is: LEADER
W20260924 23:19:47.098822 19699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:19:47.098876 19699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:19:47.098943 19689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:19:47.099112 19689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:19:47.100571 19689 catalog_manager.cc:1347] Generated new cluster ID: c84a488a154f49e8ac0f2e219ad6d049
I20260924 23:19:47.100629 19689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:19:47.117247 19689 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:19:47.117765 19689 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:19:47.128453 19689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af: Generated new TSK 0
I20260924 23:19:47.128654 19689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:19:47.131313 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45637
--builtin_ntp_servers=127.1.4.212:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 23:19:47.230369 19702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:47.230568 19702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:47.230592 19702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:47.231766 19702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:47.231807 19702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:47.231832 19702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:19:47.233172 19702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:47.233397 19702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:47.233688 19702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:47.238453 19707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:47.238464 19708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:47.238611 19702 server_base.cc:1034] running on GCE node
W20260924 23:19:47.238754 19710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:47.238932 19702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:47.239166 19702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:47.240320 19702 hybrid_clock.cc:648] HybridClock initialized: now 1790291987240305 us; error 30 us; skew 500 ppm
I20260924 23:19:47.241575 19702 webserver.cc:460] Webserver started at http://127.1.4.193:40667/ using document root <none> and password file <none>
I20260924 23:19:47.241803 19702 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:47.241852 19702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:47.241979 19702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:47.243010 19702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "5f2d64f63bce43b7bc4af8ea6129538d"
format_stamp: "Formatted at 2026-09-24 23:19:47 on dist-test-slave-s1hw"
server_key: "bf71c91bb997712820577c6111b550ce"
server_key_iv: "503d85f8ec202b1ea9ea9c409f27bf36"
server_key_version: "encryptionkey@0"
I20260924 23:19:47.243348 19702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "5f2d64f63bce43b7bc4af8ea6129538d"
format_stamp: "Formatted at 2026-09-24 23:19:47 on dist-test-slave-s1hw"
server_key: "bf71c91bb997712820577c6111b550ce"
server_key_iv: "503d85f8ec202b1ea9ea9c409f27bf36"
server_key_version: "encryptionkey@0"
I20260924 23:19:47.244661 19702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:47.245383 19716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:47.245524 19702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:47.245603 19702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "5f2d64f63bce43b7bc4af8ea6129538d"
format_stamp: "Formatted at 2026-09-24 23:19:47 on dist-test-slave-s1hw"
server_key: "bf71c91bb997712820577c6111b550ce"
server_key_iv: "503d85f8ec202b1ea9ea9c409f27bf36"
server_key_version: "encryptionkey@0"
I20260924 23:19:47.245669 19702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:47.265146 19702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:47.265483 19702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:47.265620 19702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:47.265851 19702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:47.266171 19702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:47.266207 19702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:47.266237 19702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:47.266263 19702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:47.271780 19702 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:33463
I20260924 23:19:47.271855 19829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:33463 every 8 connection(s)
I20260924 23:19:47.272162 19702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:19:47.276793 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 19702
I20260924 23:19:47.276901 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:19:47.277084 1043 external_mini_cluster.cc:1428] Setting key 955be33193bd5b020a7d564b3b9f7ae4
I20260924 23:19:47.277474 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.272875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:42403 (local address 127.1.4.254:45637)
0924 23:19:47.272942 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:47.272944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:47.273173 (+ 229us) server_negotiation.cc:407] Connection header received
0924 23:19:47.273685 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:47.273686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:47.273705 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:47.273756 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:47.274453 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.275014 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.276061 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.276270 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.276327 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.277197 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:47.277198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:47.277198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:47.277204 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:47.277222 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:47.277226 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:47.277284 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:47.277383 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 23:19:47.277414 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 23:19:47.277706 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.272869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:45637 (local address 127.1.4.193:42403)
0924 23:19:47.273052 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:47.273087 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:47.273587 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:47.273829 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:47.273833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:47.273843 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:47.274359 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:47.274365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.275107 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.275108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:47.275977 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:47.275980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.276343 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.276344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:47.276469 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.277044 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:19:47.277044 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:47.277073 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:19:47.277096 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:47.277296 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:47.277302 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:47.277306 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:19:47.277374 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 23:19:47.277480 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260924 23:19:47.278064 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45637
--builtin_ntp_servers=127.1.4.212:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 23:19:47.278234 19830 heartbeater.cc:347] Connected to a master server at 127.1.4.254:45637
I20260924 23:19:47.278327 19830 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:47.278517 19830 heartbeater.cc:511] Master 127.1.4.254:45637 requested a full tablet report, sending...
I20260924 23:19:47.279022 19642 ts_manager.cc:194] Registered new tserver with Master: 5f2d64f63bce43b7bc4af8ea6129538d (127.1.4.193:33463)
I20260924 23:19:47.279912 19642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.4.193:42403
W20260924 23:19:47.361860 19833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:47.362076 19833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:47.362098 19833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:47.363355 19833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:47.363442 19833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:47.363471 19833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:19:47.364953 19833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:47.365185 19833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:47.365480 19833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:47.370658 19841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:47.370658 19838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:47.370814 19839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:47.370860 19833 server_base.cc:1034] running on GCE node
I20260924 23:19:47.371145 19833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:47.371423 19833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:47.372602 19833 hybrid_clock.cc:648] HybridClock initialized: now 1790291987372574 us; error 48 us; skew 500 ppm
I20260924 23:19:47.374006 19833 webserver.cc:460] Webserver started at http://127.1.4.194:33349/ using document root <none> and password file <none>
I20260924 23:19:47.374264 19833 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:47.374312 19833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:47.374449 19833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:47.375782 19833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "00379bd37b2a41f59e51a8711c3b21c1"
format_stamp: "Formatted at 2026-09-24 23:19:47 on dist-test-slave-s1hw"
server_key: "7bd04758368fb39442e0b002d201cd57"
server_key_iv: "0f67279117c9683cf35b0c270bfb0fe7"
server_key_version: "encryptionkey@0"
I20260924 23:19:47.376152 19833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "00379bd37b2a41f59e51a8711c3b21c1"
format_stamp: "Formatted at 2026-09-24 23:19:47 on dist-test-slave-s1hw"
server_key: "7bd04758368fb39442e0b002d201cd57"
server_key_iv: "0f67279117c9683cf35b0c270bfb0fe7"
server_key_version: "encryptionkey@0"
I20260924 23:19:47.377465 19833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:47.378266 19847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:47.378425 19833 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:47.378502 19833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "00379bd37b2a41f59e51a8711c3b21c1"
format_stamp: "Formatted at 2026-09-24 23:19:47 on dist-test-slave-s1hw"
server_key: "7bd04758368fb39442e0b002d201cd57"
server_key_iv: "0f67279117c9683cf35b0c270bfb0fe7"
server_key_version: "encryptionkey@0"
I20260924 23:19:47.378573 19833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:47.387353 19833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:47.387748 19833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:47.387876 19833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:47.388078 19833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:47.388415 19833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:47.388451 19833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:47.388481 19833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:47.388497 19833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:47.394671 19833 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:41863
I20260924 23:19:47.394656 19960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:41863 every 8 connection(s)
I20260924 23:19:47.395082 19833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:19:47.400506 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.395830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:47399 (local address 127.1.4.254:45637)
0924 23:19:47.395881 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:47.395883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:47.396136 (+ 253us) server_negotiation.cc:407] Connection header received
0924 23:19:47.396720 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:47.396721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:47.396733 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:47.396792 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:47.397518 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.398118 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.399157 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.399346 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.399430 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.400236 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:47.400238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:47.400238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:47.400244 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:47.400260 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:47.400264 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:47.400328 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:47.400418 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 23:19:47.400441 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 23:19:47.400655 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.395820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:45637 (local address 127.1.4.194:47399)
0924 23:19:47.395979 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:47.396015 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:47.396551 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:47.396862 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:47.396866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:47.396874 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:47.397443 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:47.397449 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.398190 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.398190 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:47.399088 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:47.399091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.399434 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.399435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:47.399551 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.400075 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:19:47.400076 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:47.400107 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:19:47.400131 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:47.400333 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:47.400335 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:47.400338 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:19:47.400385 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 23:19:47.400467 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
I20260924 23:19:47.401132 19961 heartbeater.cc:347] Connected to a master server at 127.1.4.254:45637
I20260924 23:19:47.401217 19961 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:47.401394 19961 heartbeater.cc:511] Master 127.1.4.254:45637 requested a full tablet report, sending...
I20260924 23:19:47.401747 19642 ts_manager.cc:194] Registered new tserver with Master: 00379bd37b2a41f59e51a8711c3b21c1 (127.1.4.194:41863)
I20260924 23:19:47.402099 19642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.4.194:47399
I20260924 23:19:47.403518 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 19833
I20260924 23:19:47.403693 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:19:47.403906 1043 external_mini_cluster.cc:1428] Setting key 51fa6d721ca599be68ca9a28f82be77d
I20260924 23:19:47.404832 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45637
--builtin_ntp_servers=127.1.4.212:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 23:19:47.487723 19964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:47.487964 19964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:47.487987 19964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:47.489154 19964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:47.489202 19964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:47.489229 19964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:19:47.490629 19964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:45637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:47.490872 19964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:47.491176 19964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:47.496824 19972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:47.496848 19970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:47.496878 19964 server_base.cc:1034] running on GCE node
W20260924 23:19:47.496847 19969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:47.497233 19964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:47.497503 19964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:47.498667 19964 hybrid_clock.cc:648] HybridClock initialized: now 1790291987498632 us; error 49 us; skew 500 ppm
I20260924 23:19:47.500131 19964 webserver.cc:460] Webserver started at http://127.1.4.195:33755/ using document root <none> and password file <none>
I20260924 23:19:47.500348 19964 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:47.500393 19964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:47.500496 19964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:47.501554 19964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "e328b9dd02d84951890168586bdf0028"
format_stamp: "Formatted at 2026-09-24 23:19:47 on dist-test-slave-s1hw"
server_key: "1aea67e498bf2f3d91109dd3555fb2f8"
server_key_iv: "89ea0e9fb12ad951d5a69120f13d2e9a"
server_key_version: "encryptionkey@0"
I20260924 23:19:47.501871 19964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "e328b9dd02d84951890168586bdf0028"
format_stamp: "Formatted at 2026-09-24 23:19:47 on dist-test-slave-s1hw"
server_key: "1aea67e498bf2f3d91109dd3555fb2f8"
server_key_iv: "89ea0e9fb12ad951d5a69120f13d2e9a"
server_key_version: "encryptionkey@0"
I20260924 23:19:47.503051 19964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:19:47.503744 19978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:47.503887 19964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:47.503959 19964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "e328b9dd02d84951890168586bdf0028"
format_stamp: "Formatted at 2026-09-24 23:19:47 on dist-test-slave-s1hw"
server_key: "1aea67e498bf2f3d91109dd3555fb2f8"
server_key_iv: "89ea0e9fb12ad951d5a69120f13d2e9a"
server_key_version: "encryptionkey@0"
I20260924 23:19:47.504024 19964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:47.516533 19964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:47.516839 19964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:47.516954 19964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:47.517184 19964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:47.517503 19964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:47.517535 19964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:47.517566 19964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:47.517590 19964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:47.523013 19964 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:39711
I20260924 23:19:47.523057 20091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:39711 every 8 connection(s)
I20260924 23:19:47.523381 19964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:19:47.528591 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.524150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:41417 (local address 127.1.4.254:45637)
0924 23:19:47.524236 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:47.524239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:47.524440 (+ 201us) server_negotiation.cc:407] Connection header received
0924 23:19:47.525018 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:47.525019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:47.525032 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:47.525082 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:47.525758 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.526258 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.527230 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.527482 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.527541 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.528327 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:47.528329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:47.528329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:47.528334 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:47.528350 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:47.528355 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:47.528401 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:47.528480 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 23:19:47.528508 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 23:19:47.528870 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.524125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:45637 (local address 127.1.4.195:41417)
0924 23:19:47.524314 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:47.524349 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:47.524909 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:47.525156 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:47.525160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:47.525170 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:47.525658 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:47.525663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.526341 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.526342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:47.527139 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:47.527141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.527553 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.527554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:47.527685 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.528194 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:19:47.528194 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:47.528223 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:19:47.528246 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:47.528407 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:47.528412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:47.528416 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:19:47.528476 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 23:19:47.528583 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260924 23:19:47.529310 20092 heartbeater.cc:347] Connected to a master server at 127.1.4.254:45637
I20260924 23:19:47.529402 20092 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:47.529570 20092 heartbeater.cc:511] Master 127.1.4.254:45637 requested a full tablet report, sending...
I20260924 23:19:47.529561 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 19964
I20260924 23:19:47.529650 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:19:47.529836 1043 external_mini_cluster.cc:1428] Setting key 30c04dceb2950517bb3ab7f97f7598d2
I20260924 23:19:47.529948 19642 ts_manager.cc:194] Registered new tserver with Master: e328b9dd02d84951890168586bdf0028 (127.1.4.195:39711)
I20260924 23:19:47.530380 19642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.4.195:41417
I20260924 23:19:47.530807 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:19:47.534868 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.532108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.1.4.254:45637)
0924 23:19:47.532172 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:47.532174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:47.532227 (+ 53us) server_negotiation.cc:407] Connection header received
0924 23:19:47.532287 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:47.532288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:47.532296 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:47.532327 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:47.532519 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.532994 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.533738 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.533923 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.533981 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.534597 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:47.534598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:47.534598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:47.534602 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:47.534615 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:47.534618 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:47.534694 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:47.534788 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 23:19:47.534797 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 23:19:47.536532 19641 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:19:47.538021 19641 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:19:47.544493 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.540359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:39711 (local address 127.0.0.1:38958)
0924 23:19:47.540641 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:47.540649 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:47.540697 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:47.541023 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:47.541024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:47.541026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:47.541063 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:47.541064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.541888 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.541889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:47.543205 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:47.543208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.544184 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.544185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:47.544271 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.544286 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 23:19:47.544358 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 23:19:47.544411 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":35,"thread_start_us":64,"threads_started":1}
I20260924 23:19:47.544667 20101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.540103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43574 (local address 127.1.4.194:41863)
0924 23:19:47.540308 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:47.540313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:47.540323 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:19:47.540365 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:47.540366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:47.540383 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:47.540467 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:47.540833 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.541951 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.543301 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.544138 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.544195 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.544285 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:47.544534 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 23:19:47.544577 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":84,"threads_started":1}
I20260924 23:19:47.544705 20100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.540007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:41863 (local address 127.0.0.1:43574)
0924 23:19:47.540120 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:47.540160 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:47.540219 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:47.540553 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:47.540557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:47.540559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:47.540628 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:47.540630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.542023 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.542024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:47.543205 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:47.543207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.544320 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.544321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:47.544386 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.544389 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:19:47.544436 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 23:19:47.544479 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":5,"thread_start_us":49,"threads_started":1}
I20260924 23:19:47.544518 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.540467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38958 (local address 127.1.4.195:39711)
0924 23:19:47.540824 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:47.540829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:47.540838 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:19:47.540876 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:47.540878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:47.540892 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:47.540959 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:47.541135 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.541814 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.543300 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.544105 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.544188 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.544285 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:47.544378 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 23:19:47.544418 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":94,"threads_started":1}
I20260924 23:19:47.545329 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.540294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49454 (local address 127.1.4.193:33463)
0924 23:19:47.540470 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:47.540474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:47.540526 (+ 52us) server_negotiation.cc:407] Connection header received
0924 23:19:47.540608 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:47.540610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:47.540624 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:47.540695 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:47.540805 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.541951 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.544090 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.544889 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.544966 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.545055 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:47.545206 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 23:19:47.545252 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260924 23:19:47.545534 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.540213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:33463 (local address 127.0.0.1:49454)
0924 23:19:47.540473 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:47.540481 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:47.540537 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:47.540699 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:47.540699 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:47.540701 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:47.540747 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:47.540749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.543294 (+ 2545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.543295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:47.544009 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:47.544011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.544972 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.544973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:47.545090 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.545095 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:19:47.545429 (+ 334us) client_negotiation.cc:240] Negotiation successful
0924 23:19:47.545473 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":71,"threads_started":1}
I20260924 23:19:47.545528 19895 tablet_service.cc:1467] Processing CreateTablet for tablet 4e423a66f8ff4b87a6184febc0eeb8bf (DEFAULT_TABLE table=test-table [id=2f3d6b010a7441cbb11c58bc462581b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:47.545778 20026 tablet_service.cc:1467] Processing CreateTablet for tablet 4e423a66f8ff4b87a6184febc0eeb8bf (DEFAULT_TABLE table=test-table [id=2f3d6b010a7441cbb11c58bc462581b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:47.545902 19895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e423a66f8ff4b87a6184febc0eeb8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:47.546095 20026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e423a66f8ff4b87a6184febc0eeb8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:47.548460 20106 tablet_bootstrap.cc:492] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1: Bootstrap starting.
I20260924 23:19:47.549036 20107 tablet_bootstrap.cc:492] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028: Bootstrap starting.
I20260924 23:19:47.549147 20106 tablet_bootstrap.cc:654] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:47.549080 19764 tablet_service.cc:1467] Processing CreateTablet for tablet 4e423a66f8ff4b87a6184febc0eeb8bf (DEFAULT_TABLE table=test-table [id=2f3d6b010a7441cbb11c58bc462581b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:47.549391 19764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e423a66f8ff4b87a6184febc0eeb8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:47.549470 20106 log.cc:824] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:47.549651 20107 tablet_bootstrap.cc:654] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:47.549909 20107 log.cc:824] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:47.550321 20106 tablet_bootstrap.cc:492] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1: No bootstrap required, opened a new log
I20260924 23:19:47.550392 20106 ts_tablet_manager.cc:1397] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:19:47.550640 20107 tablet_bootstrap.cc:492] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028: No bootstrap required, opened a new log
I20260924 23:19:47.550705 20107 ts_tablet_manager.cc:1397] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:19:47.552273 20110 tablet_bootstrap.cc:492] T 4e423a66f8ff4b87a6184febc0eeb8bf P 5f2d64f63bce43b7bc4af8ea6129538d: Bootstrap starting.
I20260924 23:19:47.552871 20110 tablet_bootstrap.cc:654] T 4e423a66f8ff4b87a6184febc0eeb8bf P 5f2d64f63bce43b7bc4af8ea6129538d: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:47.553126 20110 log.cc:824] T 4e423a66f8ff4b87a6184febc0eeb8bf P 5f2d64f63bce43b7bc4af8ea6129538d: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:47.553105 20106 raft_consensus.cc:348] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00379bd37b2a41f59e51a8711c3b21c1" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 41863 } } peers { permanent_uuid: "5f2d64f63bce43b7bc4af8ea6129538d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33463 } } peers { permanent_uuid: "e328b9dd02d84951890168586bdf0028" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39711 } }
I20260924 23:19:47.553285 20106 raft_consensus.cc:374] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:47.553333 20106 raft_consensus.cc:729] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00379bd37b2a41f59e51a8711c3b21c1, State: Initialized, Role: FOLLOWER
I20260924 23:19:47.553336 20107 raft_consensus.cc:348] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00379bd37b2a41f59e51a8711c3b21c1" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 41863 } } peers { permanent_uuid: "5f2d64f63bce43b7bc4af8ea6129538d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33463 } } peers { permanent_uuid: "e328b9dd02d84951890168586bdf0028" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39711 } }
I20260924 23:19:47.553475 20107 raft_consensus.cc:374] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:47.553452 20106 consensus_queue.cc:260] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00379bd37b2a41f59e51a8711c3b21c1" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 41863 } } peers { permanent_uuid: "5f2d64f63bce43b7bc4af8ea6129538d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33463 } } peers { permanent_uuid: "e328b9dd02d84951890168586bdf0028" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39711 } }
I20260924 23:19:47.553511 20107 raft_consensus.cc:729] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e328b9dd02d84951890168586bdf0028, State: Initialized, Role: FOLLOWER
I20260924 23:19:47.553628 20107 consensus_queue.cc:260] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00379bd37b2a41f59e51a8711c3b21c1" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 41863 } } peers { permanent_uuid: "5f2d64f63bce43b7bc4af8ea6129538d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33463 } } peers { permanent_uuid: "e328b9dd02d84951890168586bdf0028" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39711 } }
I20260924 23:19:47.553704 20106 ts_tablet_manager.cc:1428] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 23:19:47.553859 20107 ts_tablet_manager.cc:1428] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:19:47.553871 19961 heartbeater.cc:503] Master 127.1.4.254:45637 was elected leader, sending a full tablet report...
I20260924 23:19:47.553903 20110 tablet_bootstrap.cc:492] T 4e423a66f8ff4b87a6184febc0eeb8bf P 5f2d64f63bce43b7bc4af8ea6129538d: No bootstrap required, opened a new log
I20260924 23:19:47.553959 20110 ts_tablet_manager.cc:1397] T 4e423a66f8ff4b87a6184febc0eeb8bf P 5f2d64f63bce43b7bc4af8ea6129538d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:19:47.554013 20092 heartbeater.cc:503] Master 127.1.4.254:45637 was elected leader, sending a full tablet report...
I20260924 23:19:47.556169 20110 raft_consensus.cc:348] T 4e423a66f8ff4b87a6184febc0eeb8bf P 5f2d64f63bce43b7bc4af8ea6129538d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00379bd37b2a41f59e51a8711c3b21c1" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 41863 } } peers { permanent_uuid: "5f2d64f63bce43b7bc4af8ea6129538d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33463 } } peers { permanent_uuid: "e328b9dd02d84951890168586bdf0028" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39711 } }
I20260924 23:19:47.556296 20110 raft_consensus.cc:374] T 4e423a66f8ff4b87a6184febc0eeb8bf P 5f2d64f63bce43b7bc4af8ea6129538d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:47.556316 20110 raft_consensus.cc:729] T 4e423a66f8ff4b87a6184febc0eeb8bf P 5f2d64f63bce43b7bc4af8ea6129538d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f2d64f63bce43b7bc4af8ea6129538d, State: Initialized, Role: FOLLOWER
I20260924 23:19:47.556399 20110 consensus_queue.cc:260] T 4e423a66f8ff4b87a6184febc0eeb8bf P 5f2d64f63bce43b7bc4af8ea6129538d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00379bd37b2a41f59e51a8711c3b21c1" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 41863 } } peers { permanent_uuid: "5f2d64f63bce43b7bc4af8ea6129538d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33463 } } peers { permanent_uuid: "e328b9dd02d84951890168586bdf0028" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39711 } }
I20260924 23:19:47.556612 20110 ts_tablet_manager.cc:1428] T 4e423a66f8ff4b87a6184febc0eeb8bf P 5f2d64f63bce43b7bc4af8ea6129538d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:19:47.556667 19830 heartbeater.cc:503] Master 127.1.4.254:45637 was elected leader, sending a full tablet report...
W20260924 23:19:47.645720 19962 tablet.cc:2369] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:19:47.695444 20112 raft_consensus.cc:482] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:19:47.695570 20112 raft_consensus.cc:504] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00379bd37b2a41f59e51a8711c3b21c1" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 41863 } } peers { permanent_uuid: "5f2d64f63bce43b7bc4af8ea6129538d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33463 } } peers { permanent_uuid: "e328b9dd02d84951890168586bdf0028" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39711 } }
I20260924 23:19:47.695984 20112 leader_election.cc:288] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f2d64f63bce43b7bc4af8ea6129538d (127.1.4.193:33463), e328b9dd02d84951890168586bdf0028 (127.1.4.195:39711)
I20260924 23:19:47.699087 20105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.696368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:55019 (local address 127.1.4.195:39711)
0924 23:19:47.696423 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:47.696425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:47.696519 (+ 94us) server_negotiation.cc:407] Connection header received
0924 23:19:47.696571 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:47.696572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:47.696581 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:47.696628 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:47.696870 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.697402 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.698421 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.698792 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.698861 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.698871 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:47.698992 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 23:19:47.699029 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 23:19:47.699090 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.696264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:36707 (local address 127.1.4.193:33463)
0924 23:19:47.696328 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:47.696330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:47.696341 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:19:47.696401 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:47.696402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:47.696413 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:47.696468 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:47.696724 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.697324 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.698320 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.698762 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.698814 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.698824 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:47.698971 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 23:19:47.699010 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 23:19:47.699086 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.696181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:33463 (local address 127.1.4.194:36707)
0924 23:19:47.696246 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:47.696258 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:47.696335 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:47.696487 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:47.696489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:47.696490 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:47.696562 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:47.696564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.697396 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.697397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:47.698230 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:47.698232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.698833 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.698835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:47.698902 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.698905 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:19:47.698953 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 23:19:47.698995 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 23:19:47.699101 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.696294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:39711 (local address 127.1.4.194:55019)
0924 23:19:47.696463 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:47.696471 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:47.696515 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:47.696718 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:47.696719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:47.696721 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:47.696794 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:47.696796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.697474 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.697475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:47.698347 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:47.698350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.698873 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.698875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:47.698927 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.698930 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:19:47.698983 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 23:19:47.699016 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20260924 23:19:47.699512 19784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e423a66f8ff4b87a6184febc0eeb8bf" candidate_uuid: "00379bd37b2a41f59e51a8711c3b21c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f2d64f63bce43b7bc4af8ea6129538d" is_pre_election: true
I20260924 23:19:47.699661 19784 raft_consensus.cc:2454] T 4e423a66f8ff4b87a6184febc0eeb8bf P 5f2d64f63bce43b7bc4af8ea6129538d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00379bd37b2a41f59e51a8711c3b21c1 in term 0.
I20260924 23:19:47.699640 20046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e423a66f8ff4b87a6184febc0eeb8bf" candidate_uuid: "00379bd37b2a41f59e51a8711c3b21c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e328b9dd02d84951890168586bdf0028" is_pre_election: true
I20260924 23:19:47.699777 20046 raft_consensus.cc:2454] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00379bd37b2a41f59e51a8711c3b21c1 in term 0.
I20260924 23:19:47.699878 19851 leader_election.cc:302] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00379bd37b2a41f59e51a8711c3b21c1, 5f2d64f63bce43b7bc4af8ea6129538d; no voters:
I20260924 23:19:47.699992 20112 raft_consensus.cc:2781] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:19:47.700047 20112 raft_consensus.cc:482] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:19:47.700068 20112 raft_consensus.cc:3037] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:47.700717 20112 raft_consensus.cc:504] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00379bd37b2a41f59e51a8711c3b21c1" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 41863 } } peers { permanent_uuid: "5f2d64f63bce43b7bc4af8ea6129538d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33463 } } peers { permanent_uuid: "e328b9dd02d84951890168586bdf0028" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39711 } }
I20260924 23:19:47.700835 20112 leader_election.cc:288] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f2d64f63bce43b7bc4af8ea6129538d (127.1.4.193:33463), e328b9dd02d84951890168586bdf0028 (127.1.4.195:39711)
I20260924 23:19:47.700981 19784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e423a66f8ff4b87a6184febc0eeb8bf" candidate_uuid: "00379bd37b2a41f59e51a8711c3b21c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f2d64f63bce43b7bc4af8ea6129538d"
I20260924 23:19:47.701016 20046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e423a66f8ff4b87a6184febc0eeb8bf" candidate_uuid: "00379bd37b2a41f59e51a8711c3b21c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e328b9dd02d84951890168586bdf0028"
I20260924 23:19:47.701057 19784 raft_consensus.cc:3037] T 4e423a66f8ff4b87a6184febc0eeb8bf P 5f2d64f63bce43b7bc4af8ea6129538d [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:47.701116 20046 raft_consensus.cc:3037] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:47.701764 19784 raft_consensus.cc:2454] T 4e423a66f8ff4b87a6184febc0eeb8bf P 5f2d64f63bce43b7bc4af8ea6129538d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00379bd37b2a41f59e51a8711c3b21c1 in term 1.
I20260924 23:19:47.701864 20046 raft_consensus.cc:2454] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00379bd37b2a41f59e51a8711c3b21c1 in term 1.
I20260924 23:19:47.701944 19851 leader_election.cc:302] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00379bd37b2a41f59e51a8711c3b21c1, 5f2d64f63bce43b7bc4af8ea6129538d; no voters:
I20260924 23:19:47.702034 20112 raft_consensus.cc:2781] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:47.702185 20112 raft_consensus.cc:686] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [term 1 LEADER]: Becoming Leader. State: Replica: 00379bd37b2a41f59e51a8711c3b21c1, State: Running, Role: LEADER
I20260924 23:19:47.702271 20112 consensus_queue.cc:237] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00379bd37b2a41f59e51a8711c3b21c1" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 41863 } } peers { permanent_uuid: "5f2d64f63bce43b7bc4af8ea6129538d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33463 } } peers { permanent_uuid: "e328b9dd02d84951890168586bdf0028" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39711 } }
I20260924 23:19:47.703087 19641 catalog_manager.cc:5495] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00379bd37b2a41f59e51a8711c3b21c1 (127.1.4.194). New cstate: current_term: 1 leader_uuid: "00379bd37b2a41f59e51a8711c3b21c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00379bd37b2a41f59e51a8711c3b21c1" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 41863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f2d64f63bce43b7bc4af8ea6129538d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e328b9dd02d84951890168586bdf0028" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39711 } health_report { overall_health: UNKNOWN } } }
I20260924 23:19:47.724977 20101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.722657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.1.4.194:41863)
0924 23:19:47.722718 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:47.722720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:47.722770 (+ 50us) server_negotiation.cc:407] Connection header received
0924 23:19:47.722802 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:47.722803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:47.722822 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:47.722882 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:19:47.723013 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.723544 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.724267 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:47.724454 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:47.724506 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:47.724646 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:19:47.724743 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:19:47.724827 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:47.724872 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 23:19:47.724880 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 23:19:47.726701 20046 raft_consensus.cc:1261] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028 [term 1 FOLLOWER]: Refusing update from remote peer 00379bd37b2a41f59e51a8711c3b21c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:19:47.726702 19784 raft_consensus.cc:1261] T 4e423a66f8ff4b87a6184febc0eeb8bf P 5f2d64f63bce43b7bc4af8ea6129538d [term 1 FOLLOWER]: Refusing update from remote peer 00379bd37b2a41f59e51a8711c3b21c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:19:47.727062 20112 consensus_queue.cc:1035] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e328b9dd02d84951890168586bdf0028" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 39711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:19:47.727192 20116 consensus_queue.cc:1035] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f2d64f63bce43b7bc4af8ea6129538d" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:19:47.729434 20119 mvcc.cc:205] Tried to move back new op lower bound from 7333035981725941760 to 7333035981629067264. Current Snapshot: MvccSnapshot[applied={T|T < 7333035981725941760}]
I20260924 23:19:47.729821 20120 mvcc.cc:205] Tried to move back new op lower bound from 7333035981725941760 to 7333035981629067264. Current Snapshot: MvccSnapshot[applied={T|T < 7333035981725941760}]
I20260924 23:19:47.732082 19641 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:19:47.732175 19641 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:19:47.732918 19641 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2d9f06ae515497eb54f762521a946af: Sending DeleteTablet for 3 replicas of tablet 4e423a66f8ff4b87a6184febc0eeb8bf
I20260924 23:19:47.733178 19895 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e423a66f8ff4b87a6184febc0eeb8bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:47 UTC) from {username='slave'} at 127.0.0.1:43574
I20260924 23:19:47.733285 19764 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e423a66f8ff4b87a6184febc0eeb8bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:47 UTC) from {username='slave'} at 127.0.0.1:49454
I20260924 23:19:47.733286 20026 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e423a66f8ff4b87a6184febc0eeb8bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:47 UTC) from {username='slave'} at 127.0.0.1:38958
I20260924 23:19:47.733388 20129 tablet_replica.cc:330] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1: stopping tablet replica
I20260924 23:19:47.733474 20129 raft_consensus.cc:2229] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:19:47.733590 20129 raft_consensus.cc:2258] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:19:47.733733 20131 tablet_replica.cc:330] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028: stopping tablet replica
I20260924 23:19:47.733814 20131 raft_consensus.cc:2229] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:19:47.733841 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 19702
I20260924 23:19:47.733907 20131 raft_consensus.cc:2258] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:19:47.734546 20131 ts_tablet_manager.cc:1905] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:19:47.736455 20131 ts_tablet_manager.cc:1918] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:19:47.736528 20131 log.cc:1196] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/4e423a66f8ff4b87a6184febc0eeb8bf
I20260924 23:19:47.736768 20129 ts_tablet_manager.cc:1905] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:19:47.736778 20131 ts_tablet_manager.cc:1939] T 4e423a66f8ff4b87a6184febc0eeb8bf P e328b9dd02d84951890168586bdf0028: Deleting consensus metadata
I20260924 23:19:47.737218 19629 catalog_manager.cc:4841] TS e328b9dd02d84951890168586bdf0028 (127.1.4.195:39711): tablet 4e423a66f8ff4b87a6184febc0eeb8bf (table test-table [id=2f3d6b010a7441cbb11c58bc462581b6]) successfully deleted
I20260924 23:19:47.738195 20129 ts_tablet_manager.cc:1918] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:19:47.738270 20129 log.cc:1196] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/4e423a66f8ff4b87a6184febc0eeb8bf
I20260924 23:19:47.738487 20129 ts_tablet_manager.cc:1939] T 4e423a66f8ff4b87a6184febc0eeb8bf P 00379bd37b2a41f59e51a8711c3b21c1: Deleting consensus metadata
I20260924 23:19:47.738837 19629 catalog_manager.cc:4841] TS 00379bd37b2a41f59e51a8711c3b21c1 (127.1.4.194:41863): tablet 4e423a66f8ff4b87a6184febc0eeb8bf (table test-table [id=2f3d6b010a7441cbb11c58bc462581b6]) successfully deleted
W20260924 23:19:47.740819 19629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.4.193:33463 (error 108)
I20260924 23:19:47.741006 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 19833
I20260924 23:19:47.741783 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:47.741139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:33463 (local address 127.0.0.1:49466)
0924 23:19:47.741701 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:47.741749 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538}
W20260924 23:19:47.741912 19629 catalog_manager.cc:4584] TS 5f2d64f63bce43b7bc4af8ea6129538d (127.1.4.193:33463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e423a66f8ff4b87a6184febc0eeb8bf: Network error: Client connection negotiation failed: client connection to 127.1.4.193:33463: connect: Connection refused (error 111)
I20260924 23:19:47.744993 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 19964
I20260924 23:19:47.748772 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 19611
2026-09-24T23:19:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (796 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:19:47 dist-test-slave-s1hw krb5kdc[20135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:19:47 dist-test-slave-s1hw krb5kdc[20135](info): set up 2 sockets
Sep 24 23:19:47 dist-test-slave-s1hw krb5kdc[20135](info): commencing operation
krb5kdc: starting...
W20260924 23:19:48.702958 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:19:48 dist-test-slave-s1hw krb5kdc[20135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:19:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:19:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:19:48.819100 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:40659
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:40659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:48.899410 20148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:48.899638 20148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:48.899663 20148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:48.900844 20148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:19:48.900884 20148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:48.900897 20148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:48.900908 20148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:19:48.900918 20148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:19:48.902377 20148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:40659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:40659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:48.902575 20148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:48.902765 20148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:48.907555 20156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:48.907578 20148 server_base.cc:1034] running on GCE node
W20260924 23:19:48.907539 20153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:48.907542 20154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:48.908104 20148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:48.908391 20148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:48.909550 20148 hybrid_clock.cc:648] HybridClock initialized: now 1790291988909534 us; error 29 us; skew 500 ppm
Sep 24 23:19:48 dist-test-slave-s1hw krb5kdc[20135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291988, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:48.911945 20148 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:19:48.912437 20148 webserver.cc:460] Webserver started at http://127.1.4.254:45991/ using document root <none> and password file <none>
I20260924 23:19:48.912659 20148 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:48.912709 20148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:48.912834 20148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:48.913906 20148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "dcae7b70696b4bf3905743d7e9ecffd7"
format_stamp: "Formatted at 2026-09-24 23:19:48 on dist-test-slave-s1hw"
server_key: "4fedc1d948dbe1cd77334a5e0a4ea234"
server_key_iv: "59408daa313b4d7c29d6a82f4f8abb85"
server_key_version: "encryptionkey@0"
I20260924 23:19:48.914244 20148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "dcae7b70696b4bf3905743d7e9ecffd7"
format_stamp: "Formatted at 2026-09-24 23:19:48 on dist-test-slave-s1hw"
server_key: "4fedc1d948dbe1cd77334a5e0a4ea234"
server_key_iv: "59408daa313b4d7c29d6a82f4f8abb85"
server_key_version: "encryptionkey@0"
I20260924 23:19:48.915522 20148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 23:19:48.916196 20163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:48.916373 20148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:48.916455 20148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "dcae7b70696b4bf3905743d7e9ecffd7"
format_stamp: "Formatted at 2026-09-24 23:19:48 on dist-test-slave-s1hw"
server_key: "4fedc1d948dbe1cd77334a5e0a4ea234"
server_key_iv: "59408daa313b4d7c29d6a82f4f8abb85"
server_key_version: "encryptionkey@0"
I20260924 23:19:48.916522 20148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:48.934248 20148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:48.936220 20148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:48.936360 20148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:48.940104 20148 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:40659
I20260924 23:19:48.940171 20215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:40659 every 8 connection(s)
I20260924 23:19:48.940490 20148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:19:48.941320 20216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:48.943593 20216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7: Bootstrap starting.
I20260924 23:19:48.944123 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 20148
I20260924 23:19:48.944173 20216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:48.944247 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:19:48.944456 1043 external_mini_cluster.cc:1428] Setting key 65c7ebf362f1cbe75d1960742064881e
I20260924 23:19:48.944478 20216 log.cc:824] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:48.945204 20216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7: No bootstrap required, opened a new log
I20260924 23:19:48.947544 20216 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcae7b70696b4bf3905743d7e9ecffd7" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40659 } }
I20260924 23:19:48.947723 20216 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:48.947769 20216 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcae7b70696b4bf3905743d7e9ecffd7, State: Initialized, Role: FOLLOWER
I20260924 23:19:48.947901 20216 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcae7b70696b4bf3905743d7e9ecffd7" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40659 } }
I20260924 23:19:48.947978 20216 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:19:48.948011 20216 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:19:48.948056 20216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:48.948689 20216 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcae7b70696b4bf3905743d7e9ecffd7" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40659 } }
I20260924 23:19:48.948776 20216 leader_election.cc:302] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcae7b70696b4bf3905743d7e9ecffd7; no voters:
I20260924 23:19:48.948925 20216 leader_election.cc:288] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:19:48.948962 20221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:48.949111 20221 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [term 1 LEADER]: Becoming Leader. State: Replica: dcae7b70696b4bf3905743d7e9ecffd7, State: Running, Role: LEADER
I20260924 23:19:48.949183 20216 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:19:48.949245 20221 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcae7b70696b4bf3905743d7e9ecffd7" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40659 } }
I20260924 23:19:48.949534 20221 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcae7b70696b4bf3905743d7e9ecffd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcae7b70696b4bf3905743d7e9ecffd7" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40659 } } }
I20260924 23:19:48.949604 20221 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:48.949769 20221 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcae7b70696b4bf3905743d7e9ecffd7. Latest consensus state: current_term: 1 leader_uuid: "dcae7b70696b4bf3905743d7e9ecffd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcae7b70696b4bf3905743d7e9ecffd7" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40659 } } }
I20260924 23:19:48.949887 20221 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7 [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:48.950174 20225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:19:48.950683 20225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 23:19:48 dist-test-slave-s1hw krb5kdc[20135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:19:48.952507 20225 catalog_manager.cc:1347] Generated new cluster ID: e2adf52c22164d77b54df0ec63aab636
I20260924 23:19:48.952559 20225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:19:48.954722 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:48.944959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39200 (local address 127.1.4.254:40659)
0924 23:19:48.945109 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:48.945118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:48.945142 (+ 24us) server_negotiation.cc:407] Connection header received
0924 23:19:48.945578 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:48.945586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:48.945637 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:48.945806 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:48.946531 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:48.947499 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:48.948540 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:48.949022 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:48.949123 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:48.951834 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:48.951839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:48.951840 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:48.951851 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:48.953394 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:48.953701 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:48.953705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:48.953705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:48.953792 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:48.953960 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:48.953961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:48.953961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:48.954200 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:48.954338 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:48.954438 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 23:19:48.954567 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260924 23:19:48.962482 20225 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:19:48.963024 20225 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:19:48.965128 20225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7: Generated new TSK 0
I20260924 23:19:48.965298 20225 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.4.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
Entry for principal kudu/127.1.4.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab.
I20260924 23:19:48.999845 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40659
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:49.101063 20242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:49.101297 20242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:49.101332 20242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:49.102569 20242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:49.102625 20242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:49.102663 20242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:19:49.104264 20242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.193.keytab
--principal=kudu/127.1.4.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:49.104511 20242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:49.104724 20242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:49.109793 20250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:49.109745 20248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:49.109899 20247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:49.109967 20242 server_base.cc:1034] running on GCE node
I20260924 23:19:49.110210 20242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:49.110463 20242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:49.111599 20242 hybrid_clock.cc:648] HybridClock initialized: now 1790291989111583 us; error 33 us; skew 500 ppm
Sep 24 23:19:49 dist-test-slave-s1hw krb5kdc[20135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291989, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:49.114172 20242 init.cc:377] Logged in from keytab as kudu/127.1.4.193@KRBTEST.COM (short username kudu)
I20260924 23:19:49.114616 20242 webserver.cc:460] Webserver started at http://127.1.4.193:37793/ using document root <none> and password file <none>
I20260924 23:19:49.114807 20242 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:49.114846 20242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:49.114936 20242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:49.116037 20242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "b2e3c1d9960b4e24aa292f943433ce3a"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "01f1cad98dc343bc39e63614680431e8"
server_key_iv: "5d0f3e5876d912eed58cf223a0155d56"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.116333 20242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "b2e3c1d9960b4e24aa292f943433ce3a"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "01f1cad98dc343bc39e63614680431e8"
server_key_iv: "5d0f3e5876d912eed58cf223a0155d56"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.117576 20242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:19:49.118367 20257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:49.118525 20242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:49.118610 20242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "b2e3c1d9960b4e24aa292f943433ce3a"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "01f1cad98dc343bc39e63614680431e8"
server_key_iv: "5d0f3e5876d912eed58cf223a0155d56"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.118669 20242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:49.134769 20242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:49.137080 20242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:49.137200 20242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:49.137401 20242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:49.137712 20242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:49.137748 20242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:49.137769 20242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:49.137789 20242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:49.144178 20242 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:43741
I20260924 23:19:49.144223 20370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:43741 every 8 connection(s)
I20260924 23:19:49.144574 20242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:19:49.145576 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 20242
I20260924 23:19:49.145697 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:19:49.145881 1043 external_mini_cluster.cc:1428] Setting key 2bdbe0f3a7e9699613cc1c3e422e1bc2
Sep 24 23:19:49 dist-test-slave-s1hw krb5kdc[20135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291989, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.193@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.4.194@KRBTEST.COM; defaulting to no policy
I20260924 23:19:49.154814 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.145310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:46549 (local address 127.1.4.254:40659)
0924 23:19:49.145378 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:49.145380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:49.145569 (+ 189us) server_negotiation.cc:407] Connection header received
0924 23:19:49.146250 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:49.146251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:49.146261 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:49.146310 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:49.147343 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.148051 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.149255 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.149590 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.149659 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.152126 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:49.152128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:49.152129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:49.152137 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:49.153603 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:49.154023 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:49.154025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:49.154025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:49.154083 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:49.154273 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:49.154273 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:49.154274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:49.154370 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:49.154414 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:49.154652 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 23:19:49.154744 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 23:19:49.155032 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.145281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40659 (local address 127.1.4.193:46549)
0924 23:19:49.145445 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:49.145484 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:49.146075 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:49.146403 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:49.146407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:49.147023 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:49.147242 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.147247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.148160 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.148161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.149155 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.149159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.149690 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.149691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.149806 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.150367 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:19:49.150376 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:49.152026 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:49.153681 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:49.153688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:49.153690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:49.153955 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:49.154136 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:49.154136 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:49.154137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:49.154233 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:49.154410 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:49.154413 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:49.154536 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 23:19:49.154613 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 23:19:49.154801 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260924 23:19:49.155509 20371 heartbeater.cc:347] Connected to a master server at 127.1.4.254:40659
I20260924 23:19:49.155676 20371 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:49.155872 20371 heartbeater.cc:511] Master 127.1.4.254:40659 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.194@KRBTEST.COM" created.
I20260924 23:19:49.156417 20180 ts_manager.cc:194] Registered new tserver with Master: b2e3c1d9960b4e24aa292f943433ce3a (127.1.4.193:43741)
I20260924 23:19:49.157274 20180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.193@KRBTEST.COM'} at 127.1.4.193:46549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
Entry for principal kudu/127.1.4.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab.
I20260924 23:19:49.170656 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40659
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:49.249809 20376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:49.250016 20376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:49.250047 20376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:49.251259 20376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:49.251308 20376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:49.251344 20376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:19:49.252799 20376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.194.keytab
--principal=kudu/127.1.4.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:49.253031 20376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:49.253240 20376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:49.258200 20381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:49.258208 20384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:49.258265 20376 server_base.cc:1034] running on GCE node
W20260924 23:19:49.258214 20382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:49.258621 20376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:49.258834 20376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:49.259968 20376 hybrid_clock.cc:648] HybridClock initialized: now 1790291989259955 us; error 26 us; skew 500 ppm
Sep 24 23:19:49 dist-test-slave-s1hw krb5kdc[20135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291989, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:49.262368 20376 init.cc:377] Logged in from keytab as kudu/127.1.4.194@KRBTEST.COM (short username kudu)
I20260924 23:19:49.262842 20376 webserver.cc:460] Webserver started at http://127.1.4.194:44295/ using document root <none> and password file <none>
I20260924 23:19:49.263036 20376 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:49.263077 20376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:49.263167 20376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:49.264233 20376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "47144c0f0f6945d0a575062cf4d14446"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "3aba3530e0835f918a8d3089bda11eb8"
server_key_iv: "2f0bce407b80a6ad048b6493b813a208"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.264518 20376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "47144c0f0f6945d0a575062cf4d14446"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "3aba3530e0835f918a8d3089bda11eb8"
server_key_iv: "2f0bce407b80a6ad048b6493b813a208"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.265658 20376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:19:49.266484 20391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:49.266673 20376 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:19:49.266776 20376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "47144c0f0f6945d0a575062cf4d14446"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "3aba3530e0835f918a8d3089bda11eb8"
server_key_iv: "2f0bce407b80a6ad048b6493b813a208"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.266835 20376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:49.279067 20376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:49.281340 20376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:49.281458 20376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:49.281646 20376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:49.281929 20376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:49.281960 20376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:49.281982 20376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:49.281997 20376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:49.287325 20376 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:45403
I20260924 23:19:49.287417 20504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:45403 every 8 connection(s)
I20260924 23:19:49.287731 20376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:19:49 dist-test-slave-s1hw krb5kdc[20135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291989, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.194@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:19:49.295393 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 20376
I20260924 23:19:49.295531 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:19:49.295778 1043 external_mini_cluster.cc:1428] Setting key 10901f1acaa975bba0a71aa3978b3492
I20260924 23:19:49.297025 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.288459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:54243 (local address 127.1.4.254:40659)
0924 23:19:49.288521 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:49.288523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:49.288697 (+ 174us) server_negotiation.cc:407] Connection header received
0924 23:19:49.289172 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:49.289173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:49.289201 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:49.289261 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:49.290182 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.290711 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.291710 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.291894 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.291943 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.294047 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:49.294049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:49.294049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:49.294056 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:49.295326 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:49.295903 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:49.295905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:49.295906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:49.295981 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:49.296301 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:49.296302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:49.296303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:49.296447 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:49.296528 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:49.296880 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 23:19:49.296959 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 23:19:49.297269 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.288424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40659 (local address 127.1.4.194:54243)
0924 23:19:49.288568 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:49.288611 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:49.289090 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:49.289316 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:49.289319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:49.289882 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:49.290083 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.290087 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.290812 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.290813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.291643 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.291646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.291954 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.291955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.292048 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.292547 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:19:49.292556 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:49.293961 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:49.295436 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:49.295445 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:49.295447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:49.295755 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:49.296058 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:49.296059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:49.296059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:49.296206 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:49.296527 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:49.296535 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:49.296716 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 23:19:49.296838 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 23:19:49.297007 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260924 23:19:49.297751 20505 heartbeater.cc:347] Connected to a master server at 127.1.4.254:40659
I20260924 23:19:49.297845 20505 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:49.298053 20505 heartbeater.cc:511] Master 127.1.4.254:40659 requested a full tablet report, sending...
I20260924 23:19:49.298421 20180 ts_manager.cc:194] Registered new tserver with Master: 47144c0f0f6945d0a575062cf4d14446 (127.1.4.194:45403)
I20260924 23:19:49.298805 20180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.194@KRBTEST.COM'} at 127.1.4.194:54243
WARNING: no policy specified for kudu/127.1.4.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
Entry for principal kudu/127.1.4.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab.
I20260924 23:19:49.318490 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40659
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:49.398324 20510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:49.398545 20510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:49.398569 20510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:49.399744 20510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:49.399782 20510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:49.399807 20510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:19:49.401230 20510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.195.keytab
--principal=kudu/127.1.4.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:40659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:49.401433 20510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:49.401623 20510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:49.406558 20518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:49.406594 20515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:49.406755 20510 server_base.cc:1034] running on GCE node
W20260924 23:19:49.406783 20516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:49.406955 20510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:49.407187 20510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:49.408349 20510 hybrid_clock.cc:648] HybridClock initialized: now 1790291989408331 us; error 26 us; skew 500 ppm
Sep 24 23:19:49 dist-test-slave-s1hw krb5kdc[20135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291989, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:49.410863 20510 init.cc:377] Logged in from keytab as kudu/127.1.4.195@KRBTEST.COM (short username kudu)
I20260924 23:19:49.411280 20510 webserver.cc:460] Webserver started at http://127.1.4.195:34335/ using document root <none> and password file <none>
I20260924 23:19:49.411511 20510 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:49.411566 20510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:49.411687 20510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:49.412721 20510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "de165cbd8d5149be94e95bb443a2710c"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "2240af5db90123a3a8adec7213cef68d"
server_key_iv: "5365e7520afb6b4ae3e024b958bb7fad"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.413048 20510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "de165cbd8d5149be94e95bb443a2710c"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "2240af5db90123a3a8adec7213cef68d"
server_key_iv: "5365e7520afb6b4ae3e024b958bb7fad"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.414317 20510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:19:49.415040 20525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:49.415202 20510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:49.415283 20510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "de165cbd8d5149be94e95bb443a2710c"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "2240af5db90123a3a8adec7213cef68d"
server_key_iv: "5365e7520afb6b4ae3e024b958bb7fad"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.415339 20510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:49.447263 20510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:49.449506 20510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:49.449697 20510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:49.449913 20510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:49.450189 20510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:49.450219 20510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:49.450245 20510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:49.450261 20510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:49.455870 20510 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:36327
I20260924 23:19:49.455947 20638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:36327 every 8 connection(s)
I20260924 23:19:49.456228 20510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
Sep 24 23:19:49 dist-test-slave-s1hw krb5kdc[20135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291989, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.195@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:19:49.463626 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 20510
I20260924 23:19:49.463775 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:19:49.463961 1043 external_mini_cluster.cc:1428] Setting key 086a8577932b09898287c65839e4dca7
I20260924 23:19:49.464893 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.456872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:53607 (local address 127.1.4.254:40659)
0924 23:19:49.456923 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:49.456924 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:49.457123 (+ 199us) server_negotiation.cc:407] Connection header received
0924 23:19:49.457624 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:49.457626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:49.457638 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:49.457684 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:49.458674 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.459171 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.460178 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.460375 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.460431 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.462371 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:49.462373 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:49.462373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:49.462381 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:49.463494 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:49.463901 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:49.463903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:49.463904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:49.463966 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:49.464216 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:49.464218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:49.464218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:49.464364 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:49.464428 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:49.464719 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 23:19:49.464805 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 23:19:49.465070 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.456842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40659 (local address 127.1.4.195:53607)
0924 23:19:49.457009 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:49.457044 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:49.457528 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:49.457813 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:49.457817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:49.458378 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:49.458578 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.458583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.459259 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.459260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.460091 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.460094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.460444 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.460445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.460565 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.461011 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:19:49.461019 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:49.462283 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:49.463568 (+ 1285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:49.463574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:49.463576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:49.463825 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:49.464024 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:19:49.464025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:19:49.464025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:19:49.464145 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:19:49.464425 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:49.464432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:49.464591 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 23:19:49.464686 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 23:19:49.464854 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260924 23:19:49.465513 20639 heartbeater.cc:347] Connected to a master server at 127.1.4.254:40659
I20260924 23:19:49.465677 20639 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:49.465860 20639 heartbeater.cc:511] Master 127.1.4.254:40659 requested a full tablet report, sending...
I20260924 23:19:49.466219 20180 ts_manager.cc:194] Registered new tserver with Master: de165cbd8d5149be94e95bb443a2710c (127.1.4.195:36327)
I20260924 23:19:49.466652 20180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.4.195@KRBTEST.COM'} at 127.1.4.195:53607
I20260924 23:19:49.475888 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:19:49.482755 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.477209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.1.4.254:40659)
0924 23:19:49.477268 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:49.477270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:49.477328 (+ 58us) server_negotiation.cc:407] Connection header received
0924 23:19:49.477396 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:49.477397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:49.477408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:49.477461 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:49.477997 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.478535 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.479259 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.479498 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.479549 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.480619 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:49.480620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:49.480621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:49.480630 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:49.481697 (+ 1067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:49.481992 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:49.481993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:49.481994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:49.482058 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:49.482228 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:49.482229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:49.482229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:49.482383 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:49.482473 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:49.482573 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 23:19:49.482663 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 23:19:49.484336 20180 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:19:49.485989 20180 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:19:49.492529 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.488784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.1.4.193:43741)
0924 23:19:49.488961 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:49.488966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:49.488975 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:19:49.489012 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:49.489014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:49.489027 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:49.489361 (+ 334us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:49.489439 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.490115 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.491616 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.492137 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.492204 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.492269 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:49.492398 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 23:19:49.492436 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":72,"threads_started":1}
I20260924 23:19:49.492518 20651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.488361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:43741 (local address 127.0.0.1:58990)
0924 23:19:49.488509 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:49.488551 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:49.488615 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:49.489221 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:49.489222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:49.489225 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:49.489281 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.489282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.490216 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.490217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.491517 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.491520 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.492219 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.492220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.492297 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.492310 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 23:19:49.492376 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 23:19:49.492436 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":94,"threads_started":1}
I20260924 23:19:49.492744 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.488762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36770 (local address 127.1.4.194:45403)
0924 23:19:49.488979 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:49.488982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:49.488989 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:19:49.489021 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:49.489023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:49.489038 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:49.489110 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:49.489306 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.490410 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.491473 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.492202 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.492282 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.492345 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:49.492579 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 23:19:49.492633 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
I20260924 23:19:49.492944 20650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.488566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:45403 (local address 127.0.0.1:36770)
0924 23:19:49.488778 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:49.488787 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:49.488830 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:49.489194 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:49.489195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:49.489197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:49.489236 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.489237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.490509 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.490510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.491409 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.491412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.492380 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.492381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.492444 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.492448 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:19:49.492492 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:19:49.492534 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":75,"threads_started":1}
I20260924 23:19:49.492938 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.488464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.1.4.195:36327)
0924 23:19:49.488674 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:49.488679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:49.488804 (+ 125us) server_negotiation.cc:407] Connection header received
0924 23:19:49.488939 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:49.488942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:49.488961 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:49.489041 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:49.489226 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.490379 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.491642 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.492329 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.492386 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.492434 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:49.492750 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 23:19:49.492859 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":92,"threads_started":1}
I20260924 23:19:49.493281 20653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.488361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:36327 (local address 127.0.0.1:54574)
0924 23:19:49.488735 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:49.488746 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:49.488833 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:49.489064 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:49.489067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:49.489069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:49.489144 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.489146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.490483 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.490484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.491564 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.491566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.492462 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.492463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.492516 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.492520 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:19:49.492560 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 23:19:49.493228 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":2,"thread_start_us":56,"threads_started":1}
I20260924 23:19:49.493337 20305 tablet_service.cc:1467] Processing CreateTablet for tablet bbdceb8bd9524deeaef2dc62acdaa341 (DEFAULT_TABLE table=test-table [id=dcb1850cf6f24e5eb5d4b4db05149241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:49.493741 20305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbdceb8bd9524deeaef2dc62acdaa341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:49.493981 20573 tablet_service.cc:1467] Processing CreateTablet for tablet bbdceb8bd9524deeaef2dc62acdaa341 (DEFAULT_TABLE table=test-table [id=dcb1850cf6f24e5eb5d4b4db05149241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:49.494005 20439 tablet_service.cc:1467] Processing CreateTablet for tablet bbdceb8bd9524deeaef2dc62acdaa341 (DEFAULT_TABLE table=test-table [id=dcb1850cf6f24e5eb5d4b4db05149241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:49.494268 20439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbdceb8bd9524deeaef2dc62acdaa341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:49.494290 20573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbdceb8bd9524deeaef2dc62acdaa341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:49.497292 20657 tablet_bootstrap.cc:492] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a: Bootstrap starting.
I20260924 23:19:49.497623 20658 tablet_bootstrap.cc:492] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c: Bootstrap starting.
I20260924 23:19:49.497903 20657 tablet_bootstrap.cc:654] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:49.498219 20658 tablet_bootstrap.cc:654] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:49.498404 20657 log.cc:824] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:49.498709 20658 log.cc:824] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:49.498737 20656 tablet_bootstrap.cc:492] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446: Bootstrap starting.
I20260924 23:19:49.499161 20657 tablet_bootstrap.cc:492] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a: No bootstrap required, opened a new log
I20260924 23:19:49.499233 20657 ts_tablet_manager.cc:1397] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:19:49.499383 20656 tablet_bootstrap.cc:654] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:49.499387 20658 tablet_bootstrap.cc:492] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c: No bootstrap required, opened a new log
I20260924 23:19:49.499450 20658 ts_tablet_manager.cc:1397] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:19:49.499887 20656 log.cc:824] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:49.500699 20656 tablet_bootstrap.cc:492] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446: No bootstrap required, opened a new log
I20260924 23:19:49.500769 20656 ts_tablet_manager.cc:1397] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:19:49.501017 20658 raft_consensus.cc:348] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e3c1d9960b4e24aa292f943433ce3a" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 43741 } } peers { permanent_uuid: "de165cbd8d5149be94e95bb443a2710c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36327 } } peers { permanent_uuid: "47144c0f0f6945d0a575062cf4d14446" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45403 } }
I20260924 23:19:49.501165 20658 raft_consensus.cc:374] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:49.501196 20658 raft_consensus.cc:729] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de165cbd8d5149be94e95bb443a2710c, State: Initialized, Role: FOLLOWER
I20260924 23:19:49.501186 20657 raft_consensus.cc:348] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e3c1d9960b4e24aa292f943433ce3a" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 43741 } } peers { permanent_uuid: "de165cbd8d5149be94e95bb443a2710c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36327 } } peers { permanent_uuid: "47144c0f0f6945d0a575062cf4d14446" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45403 } }
I20260924 23:19:49.501336 20658 consensus_queue.cc:260] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e3c1d9960b4e24aa292f943433ce3a" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 43741 } } peers { permanent_uuid: "de165cbd8d5149be94e95bb443a2710c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36327 } } peers { permanent_uuid: "47144c0f0f6945d0a575062cf4d14446" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45403 } }
I20260924 23:19:49.501370 20657 raft_consensus.cc:374] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:49.501406 20657 raft_consensus.cc:729] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e3c1d9960b4e24aa292f943433ce3a, State: Initialized, Role: FOLLOWER
I20260924 23:19:49.501536 20657 consensus_queue.cc:260] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e3c1d9960b4e24aa292f943433ce3a" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 43741 } } peers { permanent_uuid: "de165cbd8d5149be94e95bb443a2710c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36327 } } peers { permanent_uuid: "47144c0f0f6945d0a575062cf4d14446" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45403 } }
I20260924 23:19:49.501597 20658 ts_tablet_manager.cc:1428] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:19:49.501711 20639 heartbeater.cc:503] Master 127.1.4.254:40659 was elected leader, sending a full tablet report...
I20260924 23:19:49.501796 20657 ts_tablet_manager.cc:1428] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:19:49.501899 20371 heartbeater.cc:503] Master 127.1.4.254:40659 was elected leader, sending a full tablet report...
I20260924 23:19:49.502880 20656 raft_consensus.cc:348] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e3c1d9960b4e24aa292f943433ce3a" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 43741 } } peers { permanent_uuid: "de165cbd8d5149be94e95bb443a2710c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36327 } } peers { permanent_uuid: "47144c0f0f6945d0a575062cf4d14446" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45403 } }
I20260924 23:19:49.503024 20656 raft_consensus.cc:374] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:49.503057 20656 raft_consensus.cc:729] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47144c0f0f6945d0a575062cf4d14446, State: Initialized, Role: FOLLOWER
I20260924 23:19:49.503175 20656 consensus_queue.cc:260] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e3c1d9960b4e24aa292f943433ce3a" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 43741 } } peers { permanent_uuid: "de165cbd8d5149be94e95bb443a2710c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36327 } } peers { permanent_uuid: "47144c0f0f6945d0a575062cf4d14446" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45403 } }
I20260924 23:19:49.503451 20656 ts_tablet_manager.cc:1428] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:19:49.503492 20505 heartbeater.cc:503] Master 127.1.4.254:40659 was elected leader, sending a full tablet report...
I20260924 23:19:49.510185 20663 raft_consensus.cc:482] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:19:49.510299 20663 raft_consensus.cc:504] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e3c1d9960b4e24aa292f943433ce3a" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 43741 } } peers { permanent_uuid: "de165cbd8d5149be94e95bb443a2710c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36327 } } peers { permanent_uuid: "47144c0f0f6945d0a575062cf4d14446" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45403 } }
I20260924 23:19:49.510689 20663 leader_election.cc:288] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de165cbd8d5149be94e95bb443a2710c (127.1.4.195:36327), 47144c0f0f6945d0a575062cf4d14446 (127.1.4.194:45403)
I20260924 23:19:49.513372 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.510839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:36327 (local address 127.1.4.193:49797)
0924 23:19:49.510893 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:49.510905 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:49.510952 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:49.511128 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:49.511129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:49.511131 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:49.511203 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.511204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.511895 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.511896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.512612 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.512614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.513130 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.513131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.513213 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.513216 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:19:49.513260 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:19:49.513303 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 23:19:49.513388 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.510935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:49797 (local address 127.1.4.195:36327)
0924 23:19:49.510988 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:49.510990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:49.510999 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:19:49.511044 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:49.511044 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:49.511056 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:49.511115 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:49.511305 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.511819 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.512694 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.513061 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.513119 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.513130 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:49.513295 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 23:19:49.513335 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 23:19:49.513787 20593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbdceb8bd9524deeaef2dc62acdaa341" candidate_uuid: "b2e3c1d9960b4e24aa292f943433ce3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de165cbd8d5149be94e95bb443a2710c" is_pre_election: true
I20260924 23:19:49.513913 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.510974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:42395 (local address 127.1.4.194:45403)
0924 23:19:49.511042 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:49.511044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:49.511116 (+ 72us) server_negotiation.cc:407] Connection header received
0924 23:19:49.511194 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:49.511195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:49.511212 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:49.511259 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:49.511436 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.512053 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.513071 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.513578 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.513643 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.513654 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:49.513813 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 23:19:49.513852 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 23:19:49.513923 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.510887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:45403 (local address 127.1.4.193:42395)
0924 23:19:49.511059 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:49.511067 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:49.511138 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:49.511275 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:49.511276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:49.511278 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:49.511352 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.511353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.512156 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.512157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.512972 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.512975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.513688 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.513689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.513745 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.513749 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:19:49.513799 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 23:19:49.513838 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":101,"threads_started":1}
I20260924 23:19:49.513926 20593 raft_consensus.cc:2454] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2e3c1d9960b4e24aa292f943433ce3a in term 0.
I20260924 23:19:49.514209 20459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbdceb8bd9524deeaef2dc62acdaa341" candidate_uuid: "b2e3c1d9960b4e24aa292f943433ce3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47144c0f0f6945d0a575062cf4d14446" is_pre_election: true
I20260924 23:19:49.514308 20258 leader_election.cc:302] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e3c1d9960b4e24aa292f943433ce3a, de165cbd8d5149be94e95bb443a2710c; no voters:
I20260924 23:19:49.514370 20459 raft_consensus.cc:2454] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2e3c1d9960b4e24aa292f943433ce3a in term 0.
I20260924 23:19:49.514434 20663 raft_consensus.cc:2781] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:19:49.514508 20663 raft_consensus.cc:482] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:19:49.514544 20663 raft_consensus.cc:3037] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:49.515216 20663 raft_consensus.cc:504] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e3c1d9960b4e24aa292f943433ce3a" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 43741 } } peers { permanent_uuid: "de165cbd8d5149be94e95bb443a2710c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36327 } } peers { permanent_uuid: "47144c0f0f6945d0a575062cf4d14446" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45403 } }
I20260924 23:19:49.515439 20663 leader_election.cc:288] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [CANDIDATE]: Term 1 election: Requested vote from peers de165cbd8d5149be94e95bb443a2710c (127.1.4.195:36327), 47144c0f0f6945d0a575062cf4d14446 (127.1.4.194:45403)
I20260924 23:19:49.515534 20593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbdceb8bd9524deeaef2dc62acdaa341" candidate_uuid: "b2e3c1d9960b4e24aa292f943433ce3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de165cbd8d5149be94e95bb443a2710c"
I20260924 23:19:49.515599 20593 raft_consensus.cc:3037] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:49.515527 20459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbdceb8bd9524deeaef2dc62acdaa341" candidate_uuid: "b2e3c1d9960b4e24aa292f943433ce3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47144c0f0f6945d0a575062cf4d14446"
I20260924 23:19:49.515694 20459 raft_consensus.cc:3037] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:49.516203 20593 raft_consensus.cc:2454] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2e3c1d9960b4e24aa292f943433ce3a in term 1.
I20260924 23:19:49.516364 20258 leader_election.cc:302] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e3c1d9960b4e24aa292f943433ce3a, de165cbd8d5149be94e95bb443a2710c; no voters:
I20260924 23:19:49.516463 20663 raft_consensus.cc:2781] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:49.516503 20459 raft_consensus.cc:2454] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2e3c1d9960b4e24aa292f943433ce3a in term 1.
I20260924 23:19:49.516728 20663 raft_consensus.cc:686] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [term 1 LEADER]: Becoming Leader. State: Replica: b2e3c1d9960b4e24aa292f943433ce3a, State: Running, Role: LEADER
I20260924 23:19:49.516840 20663 consensus_queue.cc:237] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e3c1d9960b4e24aa292f943433ce3a" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 43741 } } peers { permanent_uuid: "de165cbd8d5149be94e95bb443a2710c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36327 } } peers { permanent_uuid: "47144c0f0f6945d0a575062cf4d14446" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45403 } }
I20260924 23:19:49.517688 20180 catalog_manager.cc:5495] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a reported cstate change: term changed from 0 to 1, leader changed from <none> to b2e3c1d9960b4e24aa292f943433ce3a (127.1.4.193). New cstate: current_term: 1 leader_uuid: "b2e3c1d9960b4e24aa292f943433ce3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e3c1d9960b4e24aa292f943433ce3a" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 43741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de165cbd8d5149be94e95bb443a2710c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47144c0f0f6945d0a575062cf4d14446" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45403 } health_report { overall_health: UNKNOWN } } }
I20260924 23:19:49.529855 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.527540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.1.4.193:43741)
0924 23:19:49.527602 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:49.527604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:49.527613 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:19:49.527712 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:49.527713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:49.527733 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:49.527792 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:19:49.527925 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.528410 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.529148 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.529344 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.529396 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.529543 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:19:49.529656 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:19:49.529730 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:49.529756 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 23:19:49.529775 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 23:19:49.531708 20593 raft_consensus.cc:1261] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c [term 1 FOLLOWER]: Refusing update from remote peer b2e3c1d9960b4e24aa292f943433ce3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:19:49.531709 20459 raft_consensus.cc:1261] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446 [term 1 FOLLOWER]: Refusing update from remote peer b2e3c1d9960b4e24aa292f943433ce3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:19:49.532107 20666 consensus_queue.cc:1035] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "47144c0f0f6945d0a575062cf4d14446" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:19:49.532224 20663 consensus_queue.cc:1035] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "de165cbd8d5149be94e95bb443a2710c" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 36327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:19:49.534507 20669 mvcc.cc:205] Tried to move back new op lower bound from 7333035989118758912 to 7333035989061574656. Current Snapshot: MvccSnapshot[applied={T|T < 7333035989118758912}]
I20260924 23:19:49.534515 20671 mvcc.cc:205] Tried to move back new op lower bound from 7333035989118758912 to 7333035989061574656. Current Snapshot: MvccSnapshot[applied={T|T < 7333035989118758912}]
I20260924 23:19:49.535657 20670 mvcc.cc:205] Tried to move back new op lower bound from 7333035989118758912 to 7333035989061574656. Current Snapshot: MvccSnapshot[applied={T|T < 7333035989118758912}]
I20260924 23:19:49.536795 20179 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:19:49.536880 20179 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:19:49.537662 20179 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcae7b70696b4bf3905743d7e9ecffd7: Sending DeleteTablet for 3 replicas of tablet bbdceb8bd9524deeaef2dc62acdaa341
I20260924 23:19:49.537961 20305 tablet_service.cc:1514] Processing DeleteTablet for tablet bbdceb8bd9524deeaef2dc62acdaa341 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:49 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:58990
I20260924 23:19:49.537995 20573 tablet_service.cc:1514] Processing DeleteTablet for tablet bbdceb8bd9524deeaef2dc62acdaa341 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:49 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:54574
I20260924 23:19:49.538192 20439 tablet_service.cc:1514] Processing DeleteTablet for tablet bbdceb8bd9524deeaef2dc62acdaa341 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:49 UTC) from {username='kudu', principal='kudu/127.1.4.254@KRBTEST.COM'} at 127.0.0.1:36770
I20260924 23:19:49.538255 20678 tablet_replica.cc:330] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a: stopping tablet replica
I20260924 23:19:49.538251 20679 tablet_replica.cc:330] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c: stopping tablet replica
I20260924 23:19:49.538348 20678 raft_consensus.cc:2229] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:19:49.538442 20678 raft_consensus.cc:2258] T bbdceb8bd9524deeaef2dc62acdaa341 P b2e3c1d9960b4e24aa292f943433ce3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:19:49.538523 20679 raft_consensus.cc:2229] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:19:49.538594 20680 tablet_replica.cc:330] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446: stopping tablet replica
I20260924 23:19:49.538686 20680 raft_consensus.cc:2229] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:19:49.538895 20680 raft_consensus.cc:2258] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:19:49.538946 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 20242
I20260924 23:19:49.540269 20680 ts_tablet_manager.cc:1905] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:19:49.541903 20680 ts_tablet_manager.cc:1918] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:19:49.541973 20680 log.cc:1196] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/bbdceb8bd9524deeaef2dc62acdaa341
I20260924 23:19:49.542098 20679 raft_consensus.cc:2258] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:19:49.542253 20680 ts_tablet_manager.cc:1939] T bbdceb8bd9524deeaef2dc62acdaa341 P 47144c0f0f6945d0a575062cf4d14446: Deleting consensus metadata
I20260924 23:19:49.542528 20679 ts_tablet_manager.cc:1905] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:19:49.542670 20167 catalog_manager.cc:4841] TS 47144c0f0f6945d0a575062cf4d14446 (127.1.4.194:45403): tablet bbdceb8bd9524deeaef2dc62acdaa341 (table test-table [id=dcb1850cf6f24e5eb5d4b4db05149241]) successfully deleted
I20260924 23:19:49.544059 20679 ts_tablet_manager.cc:1918] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:19:49.544118 20679 log.cc:1196] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/bbdceb8bd9524deeaef2dc62acdaa341
I20260924 23:19:49.544349 20679 ts_tablet_manager.cc:1939] T bbdceb8bd9524deeaef2dc62acdaa341 P de165cbd8d5149be94e95bb443a2710c: Deleting consensus metadata
I20260924 23:19:49.544730 20164 catalog_manager.cc:4841] TS de165cbd8d5149be94e95bb443a2710c (127.1.4.195:36327): tablet bbdceb8bd9524deeaef2dc62acdaa341 (table test-table [id=dcb1850cf6f24e5eb5d4b4db05149241]) successfully deleted
W20260924 23:19:49.545104 20167 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.4.193:43741 (error 108)
I20260924 23:19:49.545308 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 20376
I20260924 23:19:49.545938 20653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.545485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:43741 (local address 127.0.0.1:59010)
0924 23:19:49.545855 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:49.545901 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:43741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347}
W20260924 23:19:49.546203 20167 catalog_manager.cc:4584] TS b2e3c1d9960b4e24aa292f943433ce3a (127.1.4.193:43741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bbdceb8bd9524deeaef2dc62acdaa341: Network error: Client connection negotiation failed: client connection to 127.1.4.193:43741: connect: Connection refused (error 111)
I20260924 23:19:49.549652 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 20510
I20260924 23:19:49.553566 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 20148
2026-09-24T23:19:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1805 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T23:19:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:19:49Z Disabled control of system clock
I20260924 23:19:49.579075 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:41211
--webserver_interface=127.1.4.254
--webserver_port=0
--builtin_ntp_servers=127.1.4.212:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:41211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 23:19:49.661135 20685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:49.661379 20685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:49.661399 20685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:49.662540 20685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:19:49.662578 20685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:49.662590 20685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:49.662601 20685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:19:49.662621 20685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:19:49.664008 20685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:41211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:41211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:49.664208 20685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:49.664458 20685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:49.669212 20690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:49.669278 20691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:49.669353 20685 server_base.cc:1034] running on GCE node
W20260924 23:19:49.669222 20693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:49.669771 20685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:49.670058 20685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:49.671195 20685 hybrid_clock.cc:648] HybridClock initialized: now 1790291989671175 us; error 36 us; skew 500 ppm
I20260924 23:19:49.672483 20685 webserver.cc:460] Webserver started at http://127.1.4.254:44949/ using document root <none> and password file <none>
I20260924 23:19:49.672730 20685 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:49.672796 20685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:49.672909 20685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:49.673965 20685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "7a3e811f92ea4a1ebfa2cfe1fdd2153a"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "c786fcbec13c7ef339ef5bd98aba22d5"
server_key_iv: "dc6ef0bb72e44ed03f42a14f10b5d482"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.674260 20685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "7a3e811f92ea4a1ebfa2cfe1fdd2153a"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "c786fcbec13c7ef339ef5bd98aba22d5"
server_key_iv: "dc6ef0bb72e44ed03f42a14f10b5d482"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.675433 20685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:19:49.676239 20699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:49.676424 20685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:49.676522 20685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "7a3e811f92ea4a1ebfa2cfe1fdd2153a"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "c786fcbec13c7ef339ef5bd98aba22d5"
server_key_iv: "dc6ef0bb72e44ed03f42a14f10b5d482"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.676666 20685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:49.693640 20685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:49.693991 20685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:49.694137 20685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:49.698230 20685 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:41211
I20260924 23:19:49.698275 20751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:41211 every 8 connection(s)
I20260924 23:19:49.698612 20685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:19:49.699584 20752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:49.701869 20752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a: Bootstrap starting.
I20260924 23:19:49.702438 20752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:49.702699 20752 log.cc:824] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:49.703104 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 20685
I20260924 23:19:49.703212 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:19:49.703426 1043 external_mini_cluster.cc:1428] Setting key edacd694eb1654d913c571f3a09008ff
I20260924 23:19:49.703521 20752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a: No bootstrap required, opened a new log
I20260924 23:19:49.705667 20752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3e811f92ea4a1ebfa2cfe1fdd2153a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41211 } }
I20260924 23:19:49.705852 20752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:49.705886 20752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a3e811f92ea4a1ebfa2cfe1fdd2153a, State: Initialized, Role: FOLLOWER
I20260924 23:19:49.706023 20752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3e811f92ea4a1ebfa2cfe1fdd2153a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41211 } }
I20260924 23:19:49.706097 20752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:19:49.706130 20752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:19:49.706175 20752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:49.706843 20752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3e811f92ea4a1ebfa2cfe1fdd2153a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41211 } }
I20260924 23:19:49.706945 20752 leader_election.cc:302] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a3e811f92ea4a1ebfa2cfe1fdd2153a; no voters:
I20260924 23:19:49.707100 20752 leader_election.cc:288] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:19:49.707144 20757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:49.707358 20752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:19:49.707345 20757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [term 1 LEADER]: Becoming Leader. State: Replica: 7a3e811f92ea4a1ebfa2cfe1fdd2153a, State: Running, Role: LEADER
I20260924 23:19:49.708153 20757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3e811f92ea4a1ebfa2cfe1fdd2153a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41211 } }
I20260924 23:19:49.708732 20759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a3e811f92ea4a1ebfa2cfe1fdd2153a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3e811f92ea4a1ebfa2cfe1fdd2153a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41211 } } }
I20260924 23:19:49.709059 20759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:49.708873 20758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a3e811f92ea4a1ebfa2cfe1fdd2153a. Latest consensus state: current_term: 1 leader_uuid: "7a3e811f92ea4a1ebfa2cfe1fdd2153a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3e811f92ea4a1ebfa2cfe1fdd2153a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41211 } } }
I20260924 23:19:49.709226 20758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a [sys.catalog]: This master's current role is: LEADER
W20260924 23:19:49.709761 20771 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:19:49.709815 20771 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:19:49.709903 20773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:19:49.710132 20773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:19:49.710474 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.703962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36436 (local address 127.1.4.254:41211)
0924 23:19:49.704176 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:49.704185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:49.704221 (+ 36us) server_negotiation.cc:407] Connection header received
0924 23:19:49.704763 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:49.704773 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:49.704852 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:49.705001 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:49.705181 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.706994 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.708139 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.708901 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.709048 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.709743 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:49.709747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:49.709747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:49.709936 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:49.709981 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:49.709997 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:49.710055 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:49.710169 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 23:19:49.710251 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260924 23:19:49.711551 20773 catalog_manager.cc:1347] Generated new cluster ID: 1ae1a7df0a2b42d58413d88ecd0e584e
I20260924 23:19:49.711591 20773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:19:49.720332 20773 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:19:49.720861 20773 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:19:49.726625 20773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a: Generated new TSK 0
I20260924 23:19:49.726826 20773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:19:49.732281 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.193:0
--local_ip_for_outbound_sockets=127.1.4.193
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:41211
--builtin_ntp_servers=127.1.4.212:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 23:19:49.812515 20776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:49.812727 20776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:49.812750 20776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:49.813934 20776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:49.813975 20776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:49.814002 20776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.193
I20260924 23:19:49.815356 20776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.4.193
--webserver_port=0
--tserver_master_addrs=127.1.4.254:41211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.193
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:49.815614 20776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:49.815891 20776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:49.821553 20782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:49.821561 20784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:49.821561 20781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:49.821854 20776 server_base.cc:1034] running on GCE node
I20260924 23:19:49.822034 20776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:49.822281 20776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:49.823419 20776 hybrid_clock.cc:648] HybridClock initialized: now 1790291989823407 us; error 27 us; skew 500 ppm
I20260924 23:19:49.827816 20776 webserver.cc:460] Webserver started at http://127.1.4.193:44691/ using document root <none> and password file <none>
I20260924 23:19:49.828045 20776 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:49.828095 20776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:49.828213 20776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:49.829350 20776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/data/instance:
uuid: "b443446cf1e54215bb61814c06987fcf"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "cc7ebc53639ba594cc8733668b5d6a1d"
server_key_iv: "9bb28ae06933f8a7a534b8e6d3b41184"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.829676 20776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance:
uuid: "b443446cf1e54215bb61814c06987fcf"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "cc7ebc53639ba594cc8733668b5d6a1d"
server_key_iv: "9bb28ae06933f8a7a534b8e6d3b41184"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.830907 20776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:19:49.831658 20790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:49.831827 20776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:19:49.831913 20776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/wal
uuid: "b443446cf1e54215bb61814c06987fcf"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "cc7ebc53639ba594cc8733668b5d6a1d"
server_key_iv: "9bb28ae06933f8a7a534b8e6d3b41184"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.831982 20776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:49.838881 20776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:49.839156 20776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:49.839282 20776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:49.839512 20776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:49.839818 20776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:49.839850 20776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:49.839876 20776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:49.839893 20776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:49.845229 20776 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.193:33703
I20260924 23:19:49.845293 20903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.193:33703 every 8 connection(s)
I20260924 23:19:49.845582 20776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/data/info.pb
I20260924 23:19:49.846661 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 20776
I20260924 23:19:49.846763 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-0/wal/instance
I20260924 23:19:49.846925 1043 external_mini_cluster.cc:1428] Setting key e654967949b18fbee6ad194ca1774037
I20260924 23:19:49.847855 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.194:0
--local_ip_for_outbound_sockets=127.1.4.194
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:41211
--builtin_ntp_servers=127.1.4.212:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 23:19:49.851202 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.846249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.193:44069 (local address 127.1.4.254:41211)
0924 23:19:49.846297 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:49.846299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:49.846523 (+ 224us) server_negotiation.cc:407] Connection header received
0924 23:19:49.847051 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:49.847052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:49.847069 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:49.847129 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:49.847880 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.848574 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.849846 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.850067 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.850118 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.850979 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:49.850980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:49.850980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:49.850984 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:49.850999 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:49.851002 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:49.851039 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:49.851110 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 23:19:49.851132 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:19:49.851481 20906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.846235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:41211 (local address 127.1.4.193:44069)
0924 23:19:49.846390 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:49.846423 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:49.846953 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:49.847213 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:49.847217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:49.847229 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:49.847792 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.847797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.848654 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.848654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.849759 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.849762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.850142 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.850143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.850312 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.850826 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:19:49.850826 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:49.850866 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:19:49.850895 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:49.851043 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:49.851047 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:49.851050 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:19:49.851112 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 23:19:49.851228 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
I20260924 23:19:49.851970 20904 heartbeater.cc:347] Connected to a master server at 127.1.4.254:41211
I20260924 23:19:49.852069 20904 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:49.852263 20904 heartbeater.cc:511] Master 127.1.4.254:41211 requested a full tablet report, sending...
I20260924 23:19:49.852766 20716 ts_manager.cc:194] Registered new tserver with Master: b443446cf1e54215bb61814c06987fcf (127.1.4.193:33703)
I20260924 23:19:49.853370 20716 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.4.193:44069
W20260924 23:19:49.934309 20907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:49.934567 20907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:49.934599 20907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:49.936190 20907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:49.936234 20907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:49.936271 20907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.194
I20260924 23:19:49.938126 20907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.4.194
--webserver_port=0
--tserver_master_addrs=127.1.4.254:41211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.194
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:49.938408 20907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:49.938788 20907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:49.944900 20912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:49.944931 20913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:49.945194 20915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:49.945236 20907 server_base.cc:1034] running on GCE node
I20260924 23:19:49.945483 20907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:49.945734 20907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:49.946871 20907 hybrid_clock.cc:648] HybridClock initialized: now 1790291989946860 us; error 33 us; skew 500 ppm
I20260924 23:19:49.948226 20907 webserver.cc:460] Webserver started at http://127.1.4.194:40501/ using document root <none> and password file <none>
I20260924 23:19:49.948473 20907 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:49.948524 20907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:49.948653 20907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:49.949776 20907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/data/instance:
uuid: "57e51cb5b3e84f8baad883a624349326"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "068c1ccac468f46c47aa00a535dc40b8"
server_key_iv: "197e894d8ac0d0eab59f8baba1b619a3"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.950120 20907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance:
uuid: "57e51cb5b3e84f8baad883a624349326"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "068c1ccac468f46c47aa00a535dc40b8"
server_key_iv: "197e894d8ac0d0eab59f8baba1b619a3"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.951534 20907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:19:49.952363 20921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:49.952526 20907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:49.952611 20907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/wal
uuid: "57e51cb5b3e84f8baad883a624349326"
format_stamp: "Formatted at 2026-09-24 23:19:49 on dist-test-slave-s1hw"
server_key: "068c1ccac468f46c47aa00a535dc40b8"
server_key_iv: "197e894d8ac0d0eab59f8baba1b619a3"
server_key_version: "encryptionkey@0"
I20260924 23:19:49.952685 20907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:49.966686 20907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:49.967017 20907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:49.967152 20907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:49.967414 20907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:49.967767 20907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:49.967804 20907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:49.967832 20907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:49.967857 20907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:49.973397 20907 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.194:44773
I20260924 23:19:49.973417 21034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.194:44773 every 8 connection(s)
I20260924 23:19:49.973793 20907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/data/info.pb
I20260924 23:19:49.978876 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.974564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:58213 (local address 127.1.4.254:41211)
0924 23:19:49.974649 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:49.974651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:49.974856 (+ 205us) server_negotiation.cc:407] Connection header received
0924 23:19:49.975391 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:49.975392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:49.975405 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:49.975475 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:49.976137 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.976641 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.977601 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.977795 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.977855 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.978620 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:49.978621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:49.978621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:49.978625 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:49.978639 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:49.978642 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:49.978681 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:49.978773 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 23:19:49.978793 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 23:19:49.979046 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:49.974499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:41211 (local address 127.1.4.194:58213)
0924 23:19:49.974736 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:49.974771 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:49.975282 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:49.975539 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:49.975542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:49.975558 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:49.976055 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.976060 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.976726 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.976726 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.977507 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:49.977509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:49.977870 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:49.977871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:49.977984 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:49.978480 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:19:49.978481 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:49.978518 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:19:49.978547 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:49.978702 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:49.978708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:49.978711 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:19:49.978768 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 23:19:49.978851 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260924 23:19:49.979506 21035 heartbeater.cc:347] Connected to a master server at 127.1.4.254:41211
I20260924 23:19:49.979593 21035 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:49.979780 21035 heartbeater.cc:511] Master 127.1.4.254:41211 requested a full tablet report, sending...
I20260924 23:19:49.980144 20716 ts_manager.cc:194] Registered new tserver with Master: 57e51cb5b3e84f8baad883a624349326 (127.1.4.194:44773)
I20260924 23:19:49.980520 20716 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.4.194:58213
I20260924 23:19:49.982859 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 20907
I20260924 23:19:49.982940 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/wal/instance
I20260924 23:19:49.983105 1043 external_mini_cluster.cc:1428] Setting key 2ca636e0ee42de466d802a8f1ff66a92
I20260924 23:19:49.984005 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.4.195:0
--local_ip_for_outbound_sockets=127.1.4.195
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:41211
--builtin_ntp_servers=127.1.4.212:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 23:19:50.065335 21038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:50.065565 21038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:50.065587 21038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:50.066731 21038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:50.066768 21038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:50.066792 21038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.4.195
I20260924 23:19:50.068114 21038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.4.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.4.195
--webserver_port=0
--tserver_master_addrs=127.1.4.254:41211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.4.195
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:50.068354 21038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:50.068636 21038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:50.073623 21043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:50.073632 21044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:50.073635 21046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:50.074105 21038 server_base.cc:1034] running on GCE node
I20260924 23:19:50.074355 21038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:50.074591 21038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:50.075786 21038 hybrid_clock.cc:648] HybridClock initialized: now 1790291990075764 us; error 38 us; skew 500 ppm
I20260924 23:19:50.080641 21038 webserver.cc:460] Webserver started at http://127.1.4.195:33587/ using document root <none> and password file <none>
I20260924 23:19:50.080901 21038 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:50.080952 21038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:50.081075 21038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:50.082202 21038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/data/instance:
uuid: "acf83a00e583495b8802571465ab5350"
format_stamp: "Formatted at 2026-09-24 23:19:50 on dist-test-slave-s1hw"
server_key: "3d7d783ebc1d8ba7d830f6042b9e7ffc"
server_key_iv: "4e14244aafb15b7317a708c3b53739d0"
server_key_version: "encryptionkey@0"
I20260924 23:19:50.082518 21038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance:
uuid: "acf83a00e583495b8802571465ab5350"
format_stamp: "Formatted at 2026-09-24 23:19:50 on dist-test-slave-s1hw"
server_key: "3d7d783ebc1d8ba7d830f6042b9e7ffc"
server_key_iv: "4e14244aafb15b7317a708c3b53739d0"
server_key_version: "encryptionkey@0"
I20260924 23:19:50.083812 21038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:19:50.084509 21052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:50.084687 21038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:19:50.084770 21038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/wal
uuid: "acf83a00e583495b8802571465ab5350"
format_stamp: "Formatted at 2026-09-24 23:19:50 on dist-test-slave-s1hw"
server_key: "3d7d783ebc1d8ba7d830f6042b9e7ffc"
server_key_iv: "4e14244aafb15b7317a708c3b53739d0"
server_key_version: "encryptionkey@0"
I20260924 23:19:50.084829 21038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:50.097718 21038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:50.098031 21038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:50.098181 21038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:50.098424 21038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:19:50.098770 21038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:19:50.098805 21038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:50.098834 21038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:19:50.098851 21038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:50.104353 21038 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.195:38353
I20260924 23:19:50.104400 21165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.195:38353 every 8 connection(s)
I20260924 23:19:50.104736 21038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/data/info.pb
I20260924 23:19:50.108623 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 21038
I20260924 23:19:50.108736 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/wal/instance
I20260924 23:19:50.108911 1043 external_mini_cluster.cc:1428] Setting key 175752149637a18df21adc2e01b455d6
I20260924 23:19:50.110368 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.105411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:51315 (local address 127.1.4.254:41211)
0924 23:19:50.105462 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:50.105464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:50.105790 (+ 326us) server_negotiation.cc:407] Connection header received
0924 23:19:50.106305 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:50.106307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:50.106319 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:50.106370 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:50.107083 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.107623 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.108765 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.109002 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.109107 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.110104 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:50.110105 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:50.110105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:50.110110 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:50.110125 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:50.110128 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:50.110169 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:50.110272 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 23:19:50.110308 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 23:19:50.110522 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.105387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:41211 (local address 127.1.4.195:51315)
0924 23:19:50.105664 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:50.105702 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:50.106207 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:50.106450 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:50.106454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:50.106462 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:19:50.106995 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:50.107000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.107704 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.107705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:50.108666 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:50.108669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.109084 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.109085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:50.109226 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.109958 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:19:50.109959 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:19:50.109992 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:19:50.110016 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:19:50.110171 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:19:50.110173 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:19:50.110176 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:19:50.110230 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 23:19:50.110327 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260924 23:19:50.111063 21166 heartbeater.cc:347] Connected to a master server at 127.1.4.254:41211
I20260924 23:19:50.111164 21166 heartbeater.cc:464] Registering TS with master...
I20260924 23:19:50.111347 21166 heartbeater.cc:511] Master 127.1.4.254:41211 requested a full tablet report, sending...
I20260924 23:19:50.111740 20716 ts_manager.cc:194] Registered new tserver with Master: acf83a00e583495b8802571465ab5350 (127.1.4.195:38353)
I20260924 23:19:50.112095 20716 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.4.195:51315
I20260924 23:19:50.120687 1043 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:19:50.124747 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.121995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36440 (local address 127.1.4.254:41211)
0924 23:19:50.122038 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:50.122040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:50.122148 (+ 108us) server_negotiation.cc:407] Connection header received
0924 23:19:50.122226 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:50.122226 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:50.122237 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:50.122284 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:50.122441 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.122922 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.123714 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.123899 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.123951 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.124520 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:50.124521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:50.124521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:50.124526 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:50.124540 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:50.124543 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:50.124586 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:50.124643 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 23:19:50.124665 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:19:50.126129 20716 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:19:50.127736 20716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:19:50.133801 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.130139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.1.4.194:44773)
0924 23:19:50.130356 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:50.130360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:50.130369 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:19:50.130407 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:50.130409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:50.130443 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:50.130512 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:50.130690 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.131429 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.132915 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.133407 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.133471 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.133567 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:50.133663 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 23:19:50.133702 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":111,"threads_started":1}
I20260924 23:19:50.133800 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.129735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:44773 (local address 127.0.0.1:56740)
0924 23:19:50.130120 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:50.130127 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:50.130158 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:50.130578 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:50.130579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:50.130581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:50.130616 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:50.130617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.131506 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.131507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:50.132815 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:50.132817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.133487 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.133488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:50.133563 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.133577 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 23:19:50.133648 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 23:19:50.133725 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":32,"spinlock_wait_cycles":768,"thread_start_us":82,"threads_started":1}
I20260924 23:19:50.133965 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.129972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.1.4.195:38353)
0924 23:19:50.130389 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:50.130394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:50.130404 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:19:50.130438 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:50.130440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:50.130455 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:50.130530 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:50.130779 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.131784 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.132777 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.133490 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.133556 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.133644 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:50.133844 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 23:19:50.133882 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":90,"threads_started":1}
I20260924 23:19:50.134069 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.129933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33882 (local address 127.1.4.193:33703)
0924 23:19:50.130100 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:50.130105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:50.130116 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:19:50.130154 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:50.130156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:50.130171 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:50.130246 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:50.130423 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.131559 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.132896 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.133557 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.133848 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.133938 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:50.133960 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 23:19:50.133999 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260924 23:19:50.134403 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.129729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:38353 (local address 127.0.0.1:38092)
0924 23:19:50.129988 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:50.129998 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:50.130049 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:50.130649 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:50.130650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:50.130652 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:50.130694 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:50.130696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.131851 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.131852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:50.132707 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:50.132709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.133671 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.133672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:50.133730 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.133734 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:19:50.133778 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:19:50.133807 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":4,"thread_start_us":88,"threads_started":1}
I20260924 23:19:50.134407 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.129727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:33703 (local address 127.0.0.1:33882)
0924 23:19:50.129846 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:50.129890 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:50.129953 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:50.130259 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:50.130262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:50.130264 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:50.130349 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:50.130352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.131634 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.131634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:50.132814 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:50.132817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.133641 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.133642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:50.133730 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.133736 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 23:19:50.133763 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 23:19:50.133803 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":8,"thread_start_us":70,"threads_started":1}
I20260924 23:19:50.135062 20969 tablet_service.cc:1467] Processing CreateTablet for tablet 229c615f5afe44638375eda04c69e2e4 (DEFAULT_TABLE table=test-table [id=31c1672b846c422ab76beaf5d453414c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:50.135149 21100 tablet_service.cc:1467] Processing CreateTablet for tablet 229c615f5afe44638375eda04c69e2e4 (DEFAULT_TABLE table=test-table [id=31c1672b846c422ab76beaf5d453414c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:50.135442 20969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 229c615f5afe44638375eda04c69e2e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:50.135490 21100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 229c615f5afe44638375eda04c69e2e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:50.138104 21180 tablet_bootstrap.cc:492] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326: Bootstrap starting.
I20260924 23:19:50.138367 20838 tablet_service.cc:1467] Processing CreateTablet for tablet 229c615f5afe44638375eda04c69e2e4 (DEFAULT_TABLE table=test-table [id=31c1672b846c422ab76beaf5d453414c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:19:50.138707 20838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 229c615f5afe44638375eda04c69e2e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:50.138772 21180 tablet_bootstrap.cc:654] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:50.139066 21180 log.cc:824] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:50.139446 21181 tablet_bootstrap.cc:492] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350: Bootstrap starting.
I20260924 23:19:50.140049 21181 tablet_bootstrap.cc:654] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:50.140367 21181 log.cc:824] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:50.140934 21180 tablet_bootstrap.cc:492] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326: No bootstrap required, opened a new log
I20260924 23:19:50.141011 21180 ts_tablet_manager.cc:1397] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 23:19:50.141127 21181 tablet_bootstrap.cc:492] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350: No bootstrap required, opened a new log
I20260924 23:19:50.141182 21181 ts_tablet_manager.cc:1397] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:19:50.142069 21184 tablet_bootstrap.cc:492] T 229c615f5afe44638375eda04c69e2e4 P b443446cf1e54215bb61814c06987fcf: Bootstrap starting.
I20260924 23:19:50.142786 21184 tablet_bootstrap.cc:654] T 229c615f5afe44638375eda04c69e2e4 P b443446cf1e54215bb61814c06987fcf: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:50.143047 21180 raft_consensus.cc:348] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b443446cf1e54215bb61814c06987fcf" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33703 } } peers { permanent_uuid: "acf83a00e583495b8802571465ab5350" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38353 } } peers { permanent_uuid: "57e51cb5b3e84f8baad883a624349326" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44773 } }
I20260924 23:19:50.143152 21184 log.cc:824] T 229c615f5afe44638375eda04c69e2e4 P b443446cf1e54215bb61814c06987fcf: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:50.143191 21180 raft_consensus.cc:374] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:50.143224 21180 raft_consensus.cc:729] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e51cb5b3e84f8baad883a624349326, State: Initialized, Role: FOLLOWER
I20260924 23:19:50.143378 21180 consensus_queue.cc:260] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b443446cf1e54215bb61814c06987fcf" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33703 } } peers { permanent_uuid: "acf83a00e583495b8802571465ab5350" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38353 } } peers { permanent_uuid: "57e51cb5b3e84f8baad883a624349326" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44773 } }
I20260924 23:19:50.143640 21180 ts_tablet_manager.cc:1428] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:19:50.143764 21035 heartbeater.cc:503] Master 127.1.4.254:41211 was elected leader, sending a full tablet report...
I20260924 23:19:50.143657 21181 raft_consensus.cc:348] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b443446cf1e54215bb61814c06987fcf" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33703 } } peers { permanent_uuid: "acf83a00e583495b8802571465ab5350" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38353 } } peers { permanent_uuid: "57e51cb5b3e84f8baad883a624349326" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44773 } }
I20260924 23:19:50.143801 21181 raft_consensus.cc:374] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:50.143839 21181 raft_consensus.cc:729] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acf83a00e583495b8802571465ab5350, State: Initialized, Role: FOLLOWER
I20260924 23:19:50.143904 21184 tablet_bootstrap.cc:492] T 229c615f5afe44638375eda04c69e2e4 P b443446cf1e54215bb61814c06987fcf: No bootstrap required, opened a new log
I20260924 23:19:50.143968 21184 ts_tablet_manager.cc:1397] T 229c615f5afe44638375eda04c69e2e4 P b443446cf1e54215bb61814c06987fcf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:19:50.143950 21181 consensus_queue.cc:260] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b443446cf1e54215bb61814c06987fcf" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33703 } } peers { permanent_uuid: "acf83a00e583495b8802571465ab5350" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38353 } } peers { permanent_uuid: "57e51cb5b3e84f8baad883a624349326" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44773 } }
I20260924 23:19:50.144202 21181 ts_tablet_manager.cc:1428] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:19:50.144454 21166 heartbeater.cc:503] Master 127.1.4.254:41211 was elected leader, sending a full tablet report...
I20260924 23:19:50.146001 21184 raft_consensus.cc:348] T 229c615f5afe44638375eda04c69e2e4 P b443446cf1e54215bb61814c06987fcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b443446cf1e54215bb61814c06987fcf" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33703 } } peers { permanent_uuid: "acf83a00e583495b8802571465ab5350" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38353 } } peers { permanent_uuid: "57e51cb5b3e84f8baad883a624349326" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44773 } }
I20260924 23:19:50.146118 21184 raft_consensus.cc:374] T 229c615f5afe44638375eda04c69e2e4 P b443446cf1e54215bb61814c06987fcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:50.146137 21184 raft_consensus.cc:729] T 229c615f5afe44638375eda04c69e2e4 P b443446cf1e54215bb61814c06987fcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b443446cf1e54215bb61814c06987fcf, State: Initialized, Role: FOLLOWER
I20260924 23:19:50.146219 21184 consensus_queue.cc:260] T 229c615f5afe44638375eda04c69e2e4 P b443446cf1e54215bb61814c06987fcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b443446cf1e54215bb61814c06987fcf" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33703 } } peers { permanent_uuid: "acf83a00e583495b8802571465ab5350" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38353 } } peers { permanent_uuid: "57e51cb5b3e84f8baad883a624349326" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44773 } }
I20260924 23:19:50.146420 21184 ts_tablet_manager.cc:1428] T 229c615f5afe44638375eda04c69e2e4 P b443446cf1e54215bb61814c06987fcf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:19:50.146490 20904 heartbeater.cc:503] Master 127.1.4.254:41211 was elected leader, sending a full tablet report...
W20260924 23:19:50.224371 21036 tablet.cc:2369] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:19:50.346263 20905 tablet.cc:2369] T 229c615f5afe44638375eda04c69e2e4 P b443446cf1e54215bb61814c06987fcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:19:50.355234 21167 tablet.cc:2369] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:19:50.517450 21186 raft_consensus.cc:482] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:19:50.517465 21187 raft_consensus.cc:482] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:19:50.517582 21186 raft_consensus.cc:504] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b443446cf1e54215bb61814c06987fcf" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33703 } } peers { permanent_uuid: "acf83a00e583495b8802571465ab5350" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38353 } } peers { permanent_uuid: "57e51cb5b3e84f8baad883a624349326" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44773 } }
I20260924 23:19:50.517670 21187 raft_consensus.cc:504] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b443446cf1e54215bb61814c06987fcf" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33703 } } peers { permanent_uuid: "acf83a00e583495b8802571465ab5350" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38353 } } peers { permanent_uuid: "57e51cb5b3e84f8baad883a624349326" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44773 } }
I20260924 23:19:50.517956 21186 leader_election.cc:288] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b443446cf1e54215bb61814c06987fcf (127.1.4.193:33703), acf83a00e583495b8802571465ab5350 (127.1.4.195:38353)
I20260924 23:19:50.518551 21187 leader_election.cc:288] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b443446cf1e54215bb61814c06987fcf (127.1.4.193:33703), 57e51cb5b3e84f8baad883a624349326 (127.1.4.194:44773)
I20260924 23:19:50.521932 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.518385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:45597 (local address 127.1.4.195:38353)
0924 23:19:50.518420 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:50.518421 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:50.518429 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:19:50.518449 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:50.518449 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:50.518456 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:50.518501 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:50.518707 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.519495 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.521009 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.521557 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.521596 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.521608 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:50.521824 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 23:19:50.521860 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:19:50.521950 21190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.518136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:33703 (local address 127.1.4.194:33377)
0924 23:19:50.518261 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:50.518271 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:50.518323 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:50.518632 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:50.518632 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:50.518634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:50.518697 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:50.518698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.519583 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.519584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:50.520913 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:50.520915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.521722 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.521723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:50.521780 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.521782 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:19:50.521819 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 23:19:50.521860 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":78,"thread_start_us":65,"threads_started":1}
I20260924 23:19:50.522111 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.518144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:33703 (local address 127.1.4.195:53277)
0924 23:19:50.518202 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:50.518215 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:50.518258 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:50.518766 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:50.518767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:50.518769 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:50.518838 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:50.518839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.519816 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.519818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:50.521091 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:50.521094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.521886 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.521887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:50.521957 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.521961 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:19:50.521990 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 23:19:50.522034 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 23:19:50.522141 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.518519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:49387 (local address 127.1.4.194:44773)
0924 23:19:50.518551 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:50.518552 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:50.518710 (+ 158us) server_negotiation.cc:407] Connection header received
0924 23:19:50.518797 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:50.518798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:50.518806 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:50.518833 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:50.519011 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.519732 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.521189 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.521785 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.521828 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.521839 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:50.522040 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 23:19:50.522079 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:19:50.522162 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.518142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.195:38353 (local address 127.1.4.194:45597)
0924 23:19:50.518339 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:50.518349 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:50.518392 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:50.518554 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:50.518555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:50.518557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:50.518629 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:50.518631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.519584 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.519584 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:50.520912 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:50.520915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.521675 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.521676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:50.521745 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.521749 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:19:50.522043 (+ 294us) client_negotiation.cc:240] Negotiation successful
0924 23:19:50.522084 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":50,"thread_start_us":56,"threads_started":1}
I20260924 23:19:50.522342 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.518423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.194:44773 (local address 127.1.4.195:49387)
0924 23:19:50.518661 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:50.518669 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:19:50.518733 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:19:50.518886 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:19:50.518886 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:19:50.518888 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:19:50.518941 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:50.518942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.519823 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.519823 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:50.521091 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:19:50.521094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.521912 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.521913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:19:50.521971 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.521973 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:19:50.522205 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 23:19:50.522260 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":4,"thread_start_us":86,"threads_started":1}
I20260924 23:19:50.522418 21120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229c615f5afe44638375eda04c69e2e4" candidate_uuid: "57e51cb5b3e84f8baad883a624349326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acf83a00e583495b8802571465ab5350" is_pre_election: true
I20260924 23:19:50.522538 21120 raft_consensus.cc:2454] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57e51cb5b3e84f8baad883a624349326 in term 0.
I20260924 23:19:50.522688 21191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.518303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.195:53277 (local address 127.1.4.193:33703)
0924 23:19:50.518530 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:50.518532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:50.518540 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:19:50.518563 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:50.518564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:50.518573 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:50.518899 (+ 326us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:50.518937 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.519719 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.521184 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.521785 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.522525 (+ 740us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.522539 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:50.522557 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 23:19:50.522613 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":207,"thread_start_us":48,"threads_started":1}
I20260924 23:19:50.522665 20989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229c615f5afe44638375eda04c69e2e4" candidate_uuid: "acf83a00e583495b8802571465ab5350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e51cb5b3e84f8baad883a624349326" is_pre_election: true
I20260924 23:19:50.522729 20925 leader_election.cc:302] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57e51cb5b3e84f8baad883a624349326, acf83a00e583495b8802571465ab5350; no voters:
I20260924 23:19:50.522840 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.518266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.4.194:33377 (local address 127.1.4.193:33703)
0924 23:19:50.518403 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:50.518405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:50.518413 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:19:50.518442 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:50.518443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:50.518453 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:50.518598 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:19:50.518766 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.519495 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.521002 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.521541 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.521673 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.522080 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:50.522099 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 23:19:50.522138 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260924 23:19:50.522769 20989 raft_consensus.cc:2454] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acf83a00e583495b8802571465ab5350 in term 0.
I20260924 23:19:50.523015 21186 raft_consensus.cc:2781] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:19:50.522985 20858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229c615f5afe44638375eda04c69e2e4" candidate_uuid: "acf83a00e583495b8802571465ab5350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b443446cf1e54215bb61814c06987fcf" is_pre_election: true
I20260924 23:19:50.523068 21186 raft_consensus.cc:482] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:19:50.523092 21186 raft_consensus.cc:3037] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:50.523003 20857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229c615f5afe44638375eda04c69e2e4" candidate_uuid: "57e51cb5b3e84f8baad883a624349326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b443446cf1e54215bb61814c06987fcf" is_pre_election: true
I20260924 23:19:50.523178 20858 raft_consensus.cc:2454] T 229c615f5afe44638375eda04c69e2e4 P b443446cf1e54215bb61814c06987fcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acf83a00e583495b8802571465ab5350 in term 0.
I20260924 23:19:50.523402 21053 leader_election.cc:302] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acf83a00e583495b8802571465ab5350, b443446cf1e54215bb61814c06987fcf; no voters:
I20260924 23:19:50.523520 21187 raft_consensus.cc:2781] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:19:50.523595 21187 raft_consensus.cc:482] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:19:50.523618 21187 raft_consensus.cc:3037] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:50.523964 21186 raft_consensus.cc:504] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b443446cf1e54215bb61814c06987fcf" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33703 } } peers { permanent_uuid: "acf83a00e583495b8802571465ab5350" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38353 } } peers { permanent_uuid: "57e51cb5b3e84f8baad883a624349326" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44773 } }
I20260924 23:19:50.524082 21186 leader_election.cc:288] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [CANDIDATE]: Term 1 election: Requested vote from peers b443446cf1e54215bb61814c06987fcf (127.1.4.193:33703), acf83a00e583495b8802571465ab5350 (127.1.4.195:38353)
I20260924 23:19:50.524252 21120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229c615f5afe44638375eda04c69e2e4" candidate_uuid: "57e51cb5b3e84f8baad883a624349326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acf83a00e583495b8802571465ab5350"
I20260924 23:19:50.524253 20857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229c615f5afe44638375eda04c69e2e4" candidate_uuid: "57e51cb5b3e84f8baad883a624349326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b443446cf1e54215bb61814c06987fcf"
I20260924 23:19:50.524298 21187 raft_consensus.cc:504] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b443446cf1e54215bb61814c06987fcf" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33703 } } peers { permanent_uuid: "acf83a00e583495b8802571465ab5350" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38353 } } peers { permanent_uuid: "57e51cb5b3e84f8baad883a624349326" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44773 } }
I20260924 23:19:50.524327 20857 raft_consensus.cc:3037] T 229c615f5afe44638375eda04c69e2e4 P b443446cf1e54215bb61814c06987fcf [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:50.524387 21120 raft_consensus.cc:2379] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 57e51cb5b3e84f8baad883a624349326 in current term 1: Already voted for candidate acf83a00e583495b8802571465ab5350 in this term.
I20260924 23:19:50.524412 21187 leader_election.cc:288] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [CANDIDATE]: Term 1 election: Requested vote from peers b443446cf1e54215bb61814c06987fcf (127.1.4.193:33703), 57e51cb5b3e84f8baad883a624349326 (127.1.4.194:44773)
I20260924 23:19:50.524664 20858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229c615f5afe44638375eda04c69e2e4" candidate_uuid: "acf83a00e583495b8802571465ab5350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b443446cf1e54215bb61814c06987fcf"
I20260924 23:19:50.524919 20989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229c615f5afe44638375eda04c69e2e4" candidate_uuid: "acf83a00e583495b8802571465ab5350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e51cb5b3e84f8baad883a624349326"
I20260924 23:19:50.525012 20989 raft_consensus.cc:2379] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate acf83a00e583495b8802571465ab5350 in current term 1: Already voted for candidate 57e51cb5b3e84f8baad883a624349326 in this term.
I20260924 23:19:50.525173 20857 raft_consensus.cc:2454] T 229c615f5afe44638375eda04c69e2e4 P b443446cf1e54215bb61814c06987fcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57e51cb5b3e84f8baad883a624349326 in term 1.
I20260924 23:19:50.525211 21055 leader_election.cc:302] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: acf83a00e583495b8802571465ab5350; no voters: 57e51cb5b3e84f8baad883a624349326, b443446cf1e54215bb61814c06987fcf
I20260924 23:19:50.525292 21187 raft_consensus.cc:2726] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 23:19:50.525332 20922 leader_election.cc:302] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 57e51cb5b3e84f8baad883a624349326, b443446cf1e54215bb61814c06987fcf; no voters: acf83a00e583495b8802571465ab5350
I20260924 23:19:50.525403 21186 raft_consensus.cc:2781] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:50.525554 21186 raft_consensus.cc:686] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [term 1 LEADER]: Becoming Leader. State: Replica: 57e51cb5b3e84f8baad883a624349326, State: Running, Role: LEADER
I20260924 23:19:50.525635 21186 consensus_queue.cc:237] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b443446cf1e54215bb61814c06987fcf" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33703 } } peers { permanent_uuid: "acf83a00e583495b8802571465ab5350" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38353 } } peers { permanent_uuid: "57e51cb5b3e84f8baad883a624349326" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44773 } }
I20260924 23:19:50.526408 20715 catalog_manager.cc:5495] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57e51cb5b3e84f8baad883a624349326 (127.1.4.194). New cstate: current_term: 1 leader_uuid: "57e51cb5b3e84f8baad883a624349326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b443446cf1e54215bb61814c06987fcf" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acf83a00e583495b8802571465ab5350" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57e51cb5b3e84f8baad883a624349326" member_type: VOTER last_known_addr { host: "127.1.4.194" port: 44773 } health_report { overall_health: HEALTHY } } }
I20260924 23:19:50.572781 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.570468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56748 (local address 127.1.4.194:44773)
0924 23:19:50.570512 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:50.570514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:50.570523 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:19:50.570607 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:50.570608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:50.570626 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:50.570689 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:19:50.570787 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.571264 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.572067 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:50.572275 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:50.572342 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:50.572441 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:19:50.572559 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:19:50.572652 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:50.572667 (+ 15us) server_negotiation.cc:299] Negotiation successful
0924 23:19:50.572693 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:19:50.574682 21120 raft_consensus.cc:1261] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [term 1 FOLLOWER]: Refusing update from remote peer 57e51cb5b3e84f8baad883a624349326: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:19:50.574687 20857 raft_consensus.cc:1261] T 229c615f5afe44638375eda04c69e2e4 P b443446cf1e54215bb61814c06987fcf [term 1 FOLLOWER]: Refusing update from remote peer 57e51cb5b3e84f8baad883a624349326: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:19:50.574987 21193 consensus_queue.cc:1035] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b443446cf1e54215bb61814c06987fcf" member_type: VOTER last_known_addr { host: "127.1.4.193" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:19:50.575122 21186 consensus_queue.cc:1035] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acf83a00e583495b8802571465ab5350" member_type: VOTER last_known_addr { host: "127.1.4.195" port: 38353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:19:50.577870 21198 mvcc.cc:205] Tried to move back new op lower bound from 7333035993391140864 to 7333035993193598976. Current Snapshot: MvccSnapshot[applied={T|T < 7333035993391140864}]
I20260924 23:19:50.579793 20715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:19:50.579877 20715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:19:50.580523 20715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a3e811f92ea4a1ebfa2cfe1fdd2153a: Sending DeleteTablet for 3 replicas of tablet 229c615f5afe44638375eda04c69e2e4
I20260924 23:19:50.580808 20969 tablet_service.cc:1514] Processing DeleteTablet for tablet 229c615f5afe44638375eda04c69e2e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:50 UTC) from {username='slave'} at 127.0.0.1:56740
I20260924 23:19:50.580828 20838 tablet_service.cc:1514] Processing DeleteTablet for tablet 229c615f5afe44638375eda04c69e2e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:50 UTC) from {username='slave'} at 127.0.0.1:33882
I20260924 23:19:50.580876 21100 tablet_service.cc:1514] Processing DeleteTablet for tablet 229c615f5afe44638375eda04c69e2e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:19:50 UTC) from {username='slave'} at 127.0.0.1:38092
I20260924 23:19:50.581044 21206 tablet_replica.cc:330] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326: stopping tablet replica
I20260924 23:19:50.581146 21206 raft_consensus.cc:2229] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:19:50.581228 21206 raft_consensus.cc:2258] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:19:50.581350 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 20776
I20260924 23:19:50.582238 21206 ts_tablet_manager.cc:1905] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:19:50.584133 21206 ts_tablet_manager.cc:1918] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:19:50.584175 21208 tablet_replica.cc:330] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350: stopping tablet replica
I20260924 23:19:50.584301 21206 log.cc:1196] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-1/wal/wals/229c615f5afe44638375eda04c69e2e4
I20260924 23:19:50.584296 21208 raft_consensus.cc:2229] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:19:50.584405 21208 raft_consensus.cc:2258] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:19:50.584774 21206 ts_tablet_manager.cc:1939] T 229c615f5afe44638375eda04c69e2e4 P 57e51cb5b3e84f8baad883a624349326: Deleting consensus metadata
I20260924 23:19:50.585157 20702 catalog_manager.cc:4841] TS 57e51cb5b3e84f8baad883a624349326 (127.1.4.194:44773): tablet 229c615f5afe44638375eda04c69e2e4 (table test-table [id=31c1672b846c422ab76beaf5d453414c]) successfully deleted
I20260924 23:19:50.585273 21208 ts_tablet_manager.cc:1905] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:19:50.586493 21208 ts_tablet_manager.cc:1918] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:19:50.586553 21208 log.cc:1196] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350: Deleting WAL directory at /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291755455739-1043-0/minicluster-data/ts-2/wal/wals/229c615f5afe44638375eda04c69e2e4
I20260924 23:19:50.586773 21208 ts_tablet_manager.cc:1939] T 229c615f5afe44638375eda04c69e2e4 P acf83a00e583495b8802571465ab5350: Deleting consensus metadata
I20260924 23:19:50.587143 20703 catalog_manager.cc:4841] TS acf83a00e583495b8802571465ab5350 (127.1.4.195:38353): tablet 229c615f5afe44638375eda04c69e2e4 (table test-table [id=31c1672b846c422ab76beaf5d453414c]) successfully deleted
W20260924 23:19:50.588434 20700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.4.193:33703 (error 108)
I20260924 23:19:50.588562 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 20907
I20260924 23:19:50.589458 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:50.589347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.193:33703 (local address 127.0.0.1:40002)
0924 23:19:50.589395 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 23:19:50.589429 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.193:33703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 23:19:50.589579 20700 catalog_manager.cc:4584] TS b443446cf1e54215bb61814c06987fcf (127.1.4.193:33703): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 229c615f5afe44638375eda04c69e2e4: Network error: Client connection negotiation failed: client connection to 127.1.4.193:33703: connect: Connection refused (error 111)
I20260924 23:19:50.594081 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 21038
I20260924 23:19:50.598031 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 20685
2026-09-24T23:19:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1043 ms)
[----------] 6 tests from EncryptionPolicyTest (8223 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:19:50 dist-test-slave-s1hw krb5kdc[21212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:19:50 dist-test-slave-s1hw krb5kdc[21212](info): set up 2 sockets
Sep 24 23:19:50 dist-test-slave-s1hw krb5kdc[21212](info): commencing operation
krb5kdc: starting...
W20260924 23:19:51.539101 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:19:51 dist-test-slave-s1hw krb5kdc[21212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:19:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:19:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:19:51.653818 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:51.734206 21225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:51.734406 21225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:51.734437 21225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:51.735599 21225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:19:51.735639 21225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:51.735658 21225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:51.735674 21225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:19:51.735698 21225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:19:51.737134 21225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:51.737366 21225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:51.737573 21225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:51.742230 21230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:51.742201 21231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:51.742211 21233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:51.742285 21225 server_base.cc:1034] running on GCE node
I20260924 23:19:51.742843 21225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:51.743144 21225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:51.744313 21225 hybrid_clock.cc:648] HybridClock initialized: now 1790291991744281 us; error 51 us; skew 500 ppm
Sep 24 23:19:51 dist-test-slave-s1hw krb5kdc[21212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:51.746644 21225 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:19:51.747066 21225 webserver.cc:460] Webserver started at http://127.0.0.1:39385/ using document root <none> and password file <none>
I20260924 23:19:51.747285 21225 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:51.747331 21225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:51.747473 21225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:51.748560 21225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "46e560bbd580444880a86715f60fab0d"
format_stamp: "Formatted at 2026-09-24 23:19:51 on dist-test-slave-s1hw"
server_key: "7068a5253671b892a6484b1006b4e7cf"
server_key_iv: "734c87dfa3fe6df7233da62d4874093a"
server_key_version: "encryptionkey@0"
I20260924 23:19:51.748875 21225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "46e560bbd580444880a86715f60fab0d"
format_stamp: "Formatted at 2026-09-24 23:19:51 on dist-test-slave-s1hw"
server_key: "7068a5253671b892a6484b1006b4e7cf"
server_key_iv: "734c87dfa3fe6df7233da62d4874093a"
server_key_version: "encryptionkey@0"
I20260924 23:19:51.750062 21225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:19:51.750706 21240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:51.750885 21225 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:51.750964 21225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "46e560bbd580444880a86715f60fab0d"
format_stamp: "Formatted at 2026-09-24 23:19:51 on dist-test-slave-s1hw"
server_key: "7068a5253671b892a6484b1006b4e7cf"
server_key_iv: "734c87dfa3fe6df7233da62d4874093a"
server_key_version: "encryptionkey@0"
I20260924 23:19:51.751029 21225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:51.760615 21225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:51.762755 21225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:51.762878 21225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:51.766530 21225 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38981
I20260924 23:19:51.766559 21292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38981 every 8 connection(s)
I20260924 23:19:51.766886 21225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:19:51.767834 21293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:51.768127 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 21225
I20260924 23:19:51.768221 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:19:51.768402 1043 external_mini_cluster.cc:1428] Setting key 5a428f0f1c5b92b88c62613a2c9ecde5
I20260924 23:19:51.770627 21293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d: Bootstrap starting.
I20260924 23:19:51.771292 21293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:51.771611 21293 log.cc:824] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:51.772280 21293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d: No bootstrap required, opened a new log
I20260924 23:19:51.774302 21293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e560bbd580444880a86715f60fab0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38981 } }
I20260924 23:19:51.774435 21293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:51.774456 21293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e560bbd580444880a86715f60fab0d, State: Initialized, Role: FOLLOWER
I20260924 23:19:51.774566 21293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e560bbd580444880a86715f60fab0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38981 } }
I20260924 23:19:51.774639 21293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:19:51.774665 21293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:19:51.774710 21293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [term 0 FOLLOWER]: Advancing to term 1
Sep 24 23:19:51 dist-test-slave-s1hw krb5kdc[21212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 23:19:51.775256 21293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e560bbd580444880a86715f60fab0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38981 } }
I20260924 23:19:51.775341 21293 leader_election.cc:302] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46e560bbd580444880a86715f60fab0d; no voters:
I20260924 23:19:51.775521 21293 leader_election.cc:288] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:19:51.775561 21298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:51.775673 21298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [term 1 LEADER]: Becoming Leader. State: Replica: 46e560bbd580444880a86715f60fab0d, State: Running, Role: LEADER
I20260924 23:19:51.775754 21298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e560bbd580444880a86715f60fab0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38981 } }
I20260924 23:19:51.775800 21293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:19:51.776069 21298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46e560bbd580444880a86715f60fab0d. Latest consensus state: current_term: 1 leader_uuid: "46e560bbd580444880a86715f60fab0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e560bbd580444880a86715f60fab0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38981 } } }
I20260924 23:19:51.776163 21298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:51.776314 21299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46e560bbd580444880a86715f60fab0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e560bbd580444880a86715f60fab0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38981 } } }
I20260924 23:19:51.776376 21299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:51.776636 21304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:19:51.777086 21304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:19:51.778556 21304 catalog_manager.cc:1347] Generated new cluster ID: 06c58acdf4d64eaea7788378cecdb432
I20260924 23:19:51.778604 21304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:19:51.780551 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:51.768920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48477 (local address 127.0.0.1:38981)
0924 23:19:51.769109 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:51.769120 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:51.769153 (+ 33us) server_negotiation.cc:407] Connection header received
0924 23:19:51.769651 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:51.769659 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:51.769719 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:51.769855 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:51.770416 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:51.771806 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:51.772628 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:51.773219 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:51.773342 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:51.775521 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:51.775525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:51.775526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:51.775538 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:51.778862 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:51.779311 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:51.779315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:51.779315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:51.779448 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:51.779654 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:51.779655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:51.779655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:51.779978 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:51.780150 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:51.780241 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 23:19:51.780343 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260924 23:19:51.781603 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:19:51.790313 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:51.782942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34017 (local address 127.0.0.1:38981)
0924 23:19:51.782972 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:51.782974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:51.783510 (+ 536us) server_negotiation.cc:407] Connection header received
0924 23:19:51.783576 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:51.783577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:51.783586 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:51.783639 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:51.784248 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:51.784975 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:51.785980 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:51.786288 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:51.786337 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:51.787734 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:51.787736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:51.787736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:51.787743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:51.789198 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:51.789584 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:51.789584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:51.789585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:51.789638 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:51.789836 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:51.789836 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:51.789837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:51.789947 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:51.790033 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:51.790157 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 23:19:51.790214 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 23:19:51.797230 21304 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:19:51.797737 21304 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:19:51.803923 21304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46e560bbd580444880a86715f60fab0d: Generated new TSK 0
I20260924 23:19:51.804117 21304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:19:51.809440 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 21225
W20260924 23:19:51.813526 21210 connection.cc:535] client connection to 127.0.0.1:38981 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38981): Connection reset by peer (error 104)
2026-09-24T23:19:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:19:51 dist-test-slave-s1hw krb5kdc[21324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:19:51 dist-test-slave-s1hw krb5kdc[21324](info): set up 2 sockets
Sep 24 23:19:51 dist-test-slave-s1hw krb5kdc[21324](info): commencing operation
krb5kdc: starting...
W20260924 23:19:52.763083 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:19:52 dist-test-slave-s1hw krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:19:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:19:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:19:52.875705 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39065
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:52.954416 21337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:52.954620 21337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:52.954640 21337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:52.955834 21337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:19:52.955876 21337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:52.955889 21337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:52.955900 21337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:19:52.955911 21337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:19:52.957315 21337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39065
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:52.957508 21337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:52.957684 21337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:52.962255 21343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:52.962245 21345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:52.962245 21342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:52.962620 21337 server_base.cc:1034] running on GCE node
I20260924 23:19:52.962841 21337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:52.963107 21337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:52.964259 21337 hybrid_clock.cc:648] HybridClock initialized: now 1790291992964240 us; error 27 us; skew 500 ppm
Sep 24 23:19:52 dist-test-slave-s1hw krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:52.966728 21337 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:19:52.967129 21337 webserver.cc:460] Webserver started at http://127.0.0.1:36357/ using document root <none> and password file <none>
I20260924 23:19:52.967329 21337 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:52.967401 21337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:52.967523 21337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:52.968586 21337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "d2a6869899bb45b49868818cf8a211bf"
format_stamp: "Formatted at 2026-09-24 23:19:52 on dist-test-slave-s1hw"
server_key: "888de11472b130fdeebbab8c2357bb2a"
server_key_iv: "06f9cb492e2a673a697b656894b54d0b"
server_key_version: "encryptionkey@0"
I20260924 23:19:52.968910 21337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "d2a6869899bb45b49868818cf8a211bf"
format_stamp: "Formatted at 2026-09-24 23:19:52 on dist-test-slave-s1hw"
server_key: "888de11472b130fdeebbab8c2357bb2a"
server_key_iv: "06f9cb492e2a673a697b656894b54d0b"
server_key_version: "encryptionkey@0"
I20260924 23:19:52.970135 21337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:19:52.970775 21352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:52.970923 21337 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:19:52.971014 21337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "d2a6869899bb45b49868818cf8a211bf"
format_stamp: "Formatted at 2026-09-24 23:19:52 on dist-test-slave-s1hw"
server_key: "888de11472b130fdeebbab8c2357bb2a"
server_key_iv: "06f9cb492e2a673a697b656894b54d0b"
server_key_version: "encryptionkey@0"
I20260924 23:19:52.971069 21337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:52.984830 21337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:52.986544 21337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:52.986676 21337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:52.990356 21337 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39065
I20260924 23:19:52.990402 21404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39065 every 8 connection(s)
I20260924 23:19:52.990731 21337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:19:52.991674 21405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:52.993912 21405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf: Bootstrap starting.
I20260924 23:19:52.994460 21405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:52.994707 21405 log.cc:824] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:52.995354 21405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf: No bootstrap required, opened a new log
I20260924 23:19:52.996963 21405 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a6869899bb45b49868818cf8a211bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39065 } }
I20260924 23:19:52.997102 21405 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:52.997123 21405 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a6869899bb45b49868818cf8a211bf, State: Initialized, Role: FOLLOWER
I20260924 23:19:52.997253 21405 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a6869899bb45b49868818cf8a211bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39065 } }
I20260924 23:19:52.997315 21405 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:19:52.997336 21405 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:19:52.997381 21405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:52.997891 21405 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a6869899bb45b49868818cf8a211bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39065 } }
I20260924 23:19:52.997978 21405 leader_election.cc:302] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2a6869899bb45b49868818cf8a211bf; no voters:
I20260924 23:19:52.998121 21405 leader_election.cc:288] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:19:52.998167 21408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:52.998340 21405 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:19:52.998347 21408 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [term 1 LEADER]: Becoming Leader. State: Replica: d2a6869899bb45b49868818cf8a211bf, State: Running, Role: LEADER
I20260924 23:19:52.998490 21408 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a6869899bb45b49868818cf8a211bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39065 } }
I20260924 23:19:52.998929 21409 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2a6869899bb45b49868818cf8a211bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a6869899bb45b49868818cf8a211bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39065 } } }
I20260924 23:19:52.999008 21409 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:52.999054 21410 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2a6869899bb45b49868818cf8a211bf. Latest consensus state: current_term: 1 leader_uuid: "d2a6869899bb45b49868818cf8a211bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a6869899bb45b49868818cf8a211bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39065 } } }
I20260924 23:19:52.999100 21410 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:52.999226 21417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:19:52.999737 21417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:19:53.000020 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 21337
I20260924 23:19:53.000126 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:19:53.000296 1043 external_mini_cluster.cc:1428] Setting key a2a7cb3e589b1ad7c49181a6097d9100
I20260924 23:19:53.002636 21417 catalog_manager.cc:1347] Generated new cluster ID: 32f1162023d843868ff3c5087d2b055e
I20260924 23:19:53.002691 21417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:19:53.005764 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:53.000740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39591 (local address 127.0.0.1:39065)
0924 23:19:53.000900 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:53.000910 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:53.000945 (+ 35us) server_negotiation.cc:407] Connection header received
0924 23:19:53.001062 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:53.001069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:53.001124 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:53.001268 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:53.001474 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:53.002486 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:53.003617 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:53.004295 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:53.004410 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:53.005088 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:53.005092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:53.005093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:53.005296 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:53.005336 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:53.005351 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:53.005439 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:53.005505 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 23:19:53.005537 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
I20260924 23:19:53.006685 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:19:53.011585 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:53.008130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58989 (local address 127.0.0.1:39065)
0924 23:19:53.008166 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:53.008167 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:53.008319 (+ 152us) server_negotiation.cc:407] Connection header received
0924 23:19:53.008386 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:53.008387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:53.008398 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:53.008440 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:53.008545 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:53.009284 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:53.010239 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:53.010563 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:53.010630 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:53.011247 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:53.011248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:53.011249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:53.011265 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:53.011280 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:53.011284 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:53.011344 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:53.011495 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 23:19:53.011510 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:19:53.023427 21417 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:19:53.024163 21417 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:19:53.033914 21417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2a6869899bb45b49868818cf8a211bf: Generated new TSK 0
I20260924 23:19:53.034117 21417 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:19:53.040333 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 21337
W20260924 23:19:53.044284 21322 connection.cc:535] client connection to 127.0.0.1:39065 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39065): Connection reset by peer (error 104)
2026-09-24T23:19:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1231 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:19:53 dist-test-slave-s1hw krb5kdc[21436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:19:53 dist-test-slave-s1hw krb5kdc[21436](info): set up 2 sockets
Sep 24 23:19:53 dist-test-slave-s1hw krb5kdc[21436](info): commencing operation
krb5kdc: starting...
W20260924 23:19:53.995060 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:19:54 dist-test-slave-s1hw krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:19:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:19:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:19:54.106423 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:54.187929 21449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:54.188153 21449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:54.188184 21449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:54.189358 21449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:19:54.189402 21449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:54.189419 21449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:54.189435 21449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:19:54.189450 21449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:19:54.190936 21449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39997
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:54.191185 21449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:54.191419 21449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:54.196118 21455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:54.196158 21454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:54.196198 21457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:54.196234 21449 server_base.cc:1034] running on GCE node
I20260924 23:19:54.196614 21449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:54.196879 21449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:54.198014 21449 hybrid_clock.cc:648] HybridClock initialized: now 1790291994197985 us; error 39 us; skew 500 ppm
Sep 24 23:19:54 dist-test-slave-s1hw krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:54.200356 21449 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:19:54.200835 21449 webserver.cc:460] Webserver started at http://127.0.0.1:35845/ using document root <none> and password file <none>
I20260924 23:19:54.201028 21449 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:54.201068 21449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:54.201157 21449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:54.202171 21449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "d768cf0d51d2442abaa45263e187ea9d"
format_stamp: "Formatted at 2026-09-24 23:19:54 on dist-test-slave-s1hw"
server_key: "003b688235323024aa7692021bf47f95"
server_key_iv: "9795d573563d263c1b994e5079f82c2c"
server_key_version: "encryptionkey@0"
I20260924 23:19:54.202459 21449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "d768cf0d51d2442abaa45263e187ea9d"
format_stamp: "Formatted at 2026-09-24 23:19:54 on dist-test-slave-s1hw"
server_key: "003b688235323024aa7692021bf47f95"
server_key_iv: "9795d573563d263c1b994e5079f82c2c"
server_key_version: "encryptionkey@0"
I20260924 23:19:54.203715 21449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:19:54.204550 21464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:54.204795 21449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:54.204865 21449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "d768cf0d51d2442abaa45263e187ea9d"
format_stamp: "Formatted at 2026-09-24 23:19:54 on dist-test-slave-s1hw"
server_key: "003b688235323024aa7692021bf47f95"
server_key_iv: "9795d573563d263c1b994e5079f82c2c"
server_key_version: "encryptionkey@0"
I20260924 23:19:54.204939 21449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:54.208299 21449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:54.210114 21449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:54.210266 21449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:54.213842 21449 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39997
I20260924 23:19:54.213874 21516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39997 every 8 connection(s)
I20260924 23:19:54.214159 21449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:19:54.215161 21517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:54.217403 21517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d: Bootstrap starting.
I20260924 23:19:54.217947 21517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:54.218209 21517 log.cc:824] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:54.218827 21517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d: No bootstrap required, opened a new log
I20260924 23:19:54.220449 21517 raft_consensus.cc:348] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d768cf0d51d2442abaa45263e187ea9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39997 } }
I20260924 23:19:54.220588 21517 raft_consensus.cc:374] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:54.220619 21517 raft_consensus.cc:729] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d768cf0d51d2442abaa45263e187ea9d, State: Initialized, Role: FOLLOWER
I20260924 23:19:54.220714 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 21449
I20260924 23:19:54.220747 21517 consensus_queue.cc:260] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d768cf0d51d2442abaa45263e187ea9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39997 } }
I20260924 23:19:54.220805 21517 raft_consensus.cc:388] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:19:54.220834 21517 raft_consensus.cc:482] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:19:54.220861 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:19:54.220880 21517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:54.221020 1043 external_mini_cluster.cc:1428] Setting key 2a1142a81f181a0e805cb82831de55bf
I20260924 23:19:54.221503 21517 raft_consensus.cc:504] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d768cf0d51d2442abaa45263e187ea9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39997 } }
I20260924 23:19:54.221613 21517 leader_election.cc:302] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d768cf0d51d2442abaa45263e187ea9d; no voters:
I20260924 23:19:54.221776 21517 leader_election.cc:288] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:19:54.221971 21522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:54.222028 21517 sys_catalog.cc:564] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:19:54.222124 21522 raft_consensus.cc:686] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [term 1 LEADER]: Becoming Leader. State: Replica: d768cf0d51d2442abaa45263e187ea9d, State: Running, Role: LEADER
I20260924 23:19:54.222245 21522 consensus_queue.cc:237] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d768cf0d51d2442abaa45263e187ea9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39997 } }
I20260924 23:19:54.222617 21523 sys_catalog.cc:455] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d768cf0d51d2442abaa45263e187ea9d. Latest consensus state: current_term: 1 leader_uuid: "d768cf0d51d2442abaa45263e187ea9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d768cf0d51d2442abaa45263e187ea9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39997 } } }
I20260924 23:19:54.222688 21523 sys_catalog.cc:458] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:54.223160 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:54.221461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46643 (local address 127.0.0.1:39997)
0924 23:19:54.221653 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:54.221661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:54.221694 (+ 33us) server_negotiation.cc:407] Connection header received
0924 23:19:54.221798 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:54.221803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:54.221833 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:54.221974 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:54.222591 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:54.222595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:54.222595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:54.222599 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:54.222644 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:54.222659 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:54.222711 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:54.222938 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 23:19:54.222982 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260924 23:19:54.223179 21526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:19:54.223690 21526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:19:54.224128 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:19:54.222890 21524 sys_catalog.cc:455] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d768cf0d51d2442abaa45263e187ea9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d768cf0d51d2442abaa45263e187ea9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39997 } } }
I20260924 23:19:54.224294 21524 sys_catalog.cc:458] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:54.225163 21526 catalog_manager.cc:1347] Generated new cluster ID: 427f5116506f4159bd8e7f00572fd805
I20260924 23:19:54.225214 21526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:19:54.232760 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:54.231686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57251 (local address 127.0.0.1:39997)
0924 23:19:54.231724 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:54.231726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:54.231865 (+ 139us) server_negotiation.cc:407] Connection header received
0924 23:19:54.231927 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:54.231928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:54.231935 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:54.231979 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:54.232542 (+ 563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:54.232543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:54.232543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:54.232548 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:54.232563 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:54.232568 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:54.232615 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:54.232666 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 23:19:54.232679 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:19:54.241202 21526 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:19:54.241371 21526 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:19:54.243819 21526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d768cf0d51d2442abaa45263e187ea9d: Generated new TSK 0
I20260924 23:19:54.243983 21526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:19:54.247701 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 21449
2026-09-24T23:19:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:19:54 dist-test-slave-s1hw krb5kdc[21548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:19:54 dist-test-slave-s1hw krb5kdc[21548](info): set up 2 sockets
Sep 24 23:19:54 dist-test-slave-s1hw krb5kdc[21548](info): commencing operation
krb5kdc: starting...
W20260924 23:19:55.195420 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:19:55 dist-test-slave-s1hw krb5kdc[21548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:19:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:19:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:19:55.306772 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:55.385349 21561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:55.385552 21561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:55.385577 21561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:55.386699 21561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:19:55.386734 21561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:55.386746 21561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:55.386758 21561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:19:55.386768 21561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:19:55.388252 21561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:55.388451 21561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:55.388631 21561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:19:55.393373 21561 server_base.cc:1034] running on GCE node
W20260924 23:19:55.393381 21566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:55.393347 21567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:55.393347 21569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:55.393821 21561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:55.394078 21561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:55.395224 21561 hybrid_clock.cc:648] HybridClock initialized: now 1790291995395214 us; error 35 us; skew 500 ppm
Sep 24 23:19:55 dist-test-slave-s1hw krb5kdc[21548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:55.402078 21561 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:19:55.402515 21561 webserver.cc:460] Webserver started at http://127.0.0.1:41975/ using document root <none> and password file <none>
I20260924 23:19:55.402755 21561 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:55.402806 21561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:55.402931 21561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:55.404036 21561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "e43858650bd441b7bde163e70acb74ef"
format_stamp: "Formatted at 2026-09-24 23:19:55 on dist-test-slave-s1hw"
server_key: "22e3d9ac81356dccccbdb79942740d95"
server_key_iv: "7473fe3780eed1ef181f1ec0c465219c"
server_key_version: "encryptionkey@0"
I20260924 23:19:55.404374 21561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "e43858650bd441b7bde163e70acb74ef"
format_stamp: "Formatted at 2026-09-24 23:19:55 on dist-test-slave-s1hw"
server_key: "22e3d9ac81356dccccbdb79942740d95"
server_key_iv: "7473fe3780eed1ef181f1ec0c465219c"
server_key_version: "encryptionkey@0"
I20260924 23:19:55.405704 21561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:19:55.406424 21576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:55.406656 21561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:55.406744 21561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "e43858650bd441b7bde163e70acb74ef"
format_stamp: "Formatted at 2026-09-24 23:19:55 on dist-test-slave-s1hw"
server_key: "22e3d9ac81356dccccbdb79942740d95"
server_key_iv: "7473fe3780eed1ef181f1ec0c465219c"
server_key_version: "encryptionkey@0"
I20260924 23:19:55.406804 21561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:55.420500 21561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:55.422717 21561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:55.422855 21561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:55.426714 21561 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40197
I20260924 23:19:55.426805 21628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40197 every 8 connection(s)
I20260924 23:19:55.427103 21561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:19:55.428160 21629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:55.430466 21629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef: Bootstrap starting.
I20260924 23:19:55.431010 21629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:55.431274 21629 log.cc:824] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:55.431290 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 21561
I20260924 23:19:55.431430 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:19:55.431619 1043 external_mini_cluster.cc:1428] Setting key 08c9f386ab1f47e6e6979db3685e27bf
I20260924 23:19:55.432117 21629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef: No bootstrap required, opened a new log
I20260924 23:19:55.434265 21629 raft_consensus.cc:348] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e43858650bd441b7bde163e70acb74ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40197 } }
I20260924 23:19:55.434437 21629 raft_consensus.cc:374] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:55.434473 21629 raft_consensus.cc:729] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e43858650bd441b7bde163e70acb74ef, State: Initialized, Role: FOLLOWER
I20260924 23:19:55.434605 21629 consensus_queue.cc:260] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e43858650bd441b7bde163e70acb74ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40197 } }
I20260924 23:19:55.434687 21629 raft_consensus.cc:388] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:19:55.434728 21629 raft_consensus.cc:482] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:19:55.434777 21629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:55.435509 21629 raft_consensus.cc:504] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e43858650bd441b7bde163e70acb74ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40197 } }
I20260924 23:19:55.435616 21629 leader_election.cc:302] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e43858650bd441b7bde163e70acb74ef; no voters:
I20260924 23:19:55.435765 21629 leader_election.cc:288] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:19:55.435798 21634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:55.435974 21634 raft_consensus.cc:686] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [term 1 LEADER]: Becoming Leader. State: Replica: e43858650bd441b7bde163e70acb74ef, State: Running, Role: LEADER
I20260924 23:19:55.436005 21629 sys_catalog.cc:564] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:19:55.436169 21634 consensus_queue.cc:237] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e43858650bd441b7bde163e70acb74ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40197 } }
I20260924 23:19:55.436677 21635 sys_catalog.cc:455] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e43858650bd441b7bde163e70acb74ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e43858650bd441b7bde163e70acb74ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40197 } } }
I20260924 23:19:55.436760 21635 sys_catalog.cc:458] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:55.436959 21636 sys_catalog.cc:455] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader e43858650bd441b7bde163e70acb74ef. Latest consensus state: current_term: 1 leader_uuid: "e43858650bd441b7bde163e70acb74ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e43858650bd441b7bde163e70acb74ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40197 } } }
I20260924 23:19:55.437098 21636 sys_catalog.cc:458] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:55.437263 21641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:19:55.437780 21641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:19:55.439551 21641 catalog_manager.cc:1347] Generated new cluster ID: 1fdbbe4e0741475c946521d88f111bcf
I20260924 23:19:55.439611 21641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:19:55 dist-test-slave-s1hw krb5kdc[21548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 23:19:55.444020 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:55.432174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37985 (local address 127.0.0.1:40197)
0924 23:19:55.432354 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:55.432363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:55.432398 (+ 35us) server_negotiation.cc:407] Connection header received
0924 23:19:55.433462 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:55.433474 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:55.433541 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:55.433792 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:55.434380 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:55.436354 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:55.437471 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:55.438040 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:55.438144 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:55.440796 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:55.440800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:55.440801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:55.440813 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:55.442334 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:55.442667 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:55.442669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:55.442669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:55.442751 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:55.442923 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:55.442924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:55.442924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:55.443157 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:55.443708 (+ 551us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:55.443768 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 23:19:55.443868 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260924 23:19:55.444861 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:19:55.457216 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:55.447736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47677 (local address 127.0.0.1:40197)
0924 23:19:55.447771 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:55.447773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:55.451570 (+ 3797us) server_negotiation.cc:407] Connection header received
0924 23:19:55.451673 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:55.451674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:55.451685 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:55.451731 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:55.452288 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:55.452808 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:55.453484 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:55.453703 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:55.453758 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:55.454927 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:55.454929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:55.454929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:55.454937 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:55.456225 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:55.456531 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:55.456532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:55.456532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:55.456584 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:55.456750 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:55.456751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:55.456751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:55.456861 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:55.456942 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:55.457047 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 23:19:55.457121 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:19:55.479390 21641 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:19:55.480170 21641 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:19:55.493017 21641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e43858650bd441b7bde163e70acb74ef: Generated new TSK 0
I20260924 23:19:55.493259 21641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:19:55.501506 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 21561
2026-09-24T23:19:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1254 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:19:55 dist-test-slave-s1hw krb5kdc[21660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:19:55 dist-test-slave-s1hw krb5kdc[21660](info): set up 2 sockets
Sep 24 23:19:55 dist-test-slave-s1hw krb5kdc[21660](info): commencing operation
krb5kdc: starting...
W20260924 23:19:56.451195 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:19:56 dist-test-slave-s1hw krb5kdc[21660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:19:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:19:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:19:56.565929 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35925
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:56.646317 21673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:56.646521 21673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:56.646544 21673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:56.647703 21673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:19:56.647743 21673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:56.647755 21673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:56.647766 21673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:19:56.647776 21673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:19:56.649236 21673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:56.649434 21673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:56.649609 21673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:56.654268 21679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:56.654304 21681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:56.654361 21673 server_base.cc:1034] running on GCE node
W20260924 23:19:56.654278 21678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:56.654763 21673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:56.655050 21673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:56.656224 21673 hybrid_clock.cc:648] HybridClock initialized: now 1790291996656199 us; error 35 us; skew 500 ppm
Sep 24 23:19:56 dist-test-slave-s1hw krb5kdc[21660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:56.658872 21673 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:19:56.659282 21673 webserver.cc:460] Webserver started at http://127.0.0.1:45059/ using document root <none> and password file <none>
I20260924 23:19:56.659507 21673 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:56.659565 21673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:56.659689 21673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:56.660761 21673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "9eb092e8c2b44b4399af7e8e7a887cbe"
format_stamp: "Formatted at 2026-09-24 23:19:56 on dist-test-slave-s1hw"
server_key: "d7e8a9d9fb92eea6552aa88916f39663"
server_key_iv: "a9cb5aa92414bdba067346b94d4ce7b2"
server_key_version: "encryptionkey@0"
I20260924 23:19:56.661109 21673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "9eb092e8c2b44b4399af7e8e7a887cbe"
format_stamp: "Formatted at 2026-09-24 23:19:56 on dist-test-slave-s1hw"
server_key: "d7e8a9d9fb92eea6552aa88916f39663"
server_key_iv: "a9cb5aa92414bdba067346b94d4ce7b2"
server_key_version: "encryptionkey@0"
I20260924 23:19:56.662724 21673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:19:56.663483 21688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:56.663728 21673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:56.663823 21673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "9eb092e8c2b44b4399af7e8e7a887cbe"
format_stamp: "Formatted at 2026-09-24 23:19:56 on dist-test-slave-s1hw"
server_key: "d7e8a9d9fb92eea6552aa88916f39663"
server_key_iv: "a9cb5aa92414bdba067346b94d4ce7b2"
server_key_version: "encryptionkey@0"
I20260924 23:19:56.663897 21673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:56.670878 21673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:56.672935 21673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:56.673076 21673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:56.677021 21673 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35925
I20260924 23:19:56.677053 21740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35925 every 8 connection(s)
I20260924 23:19:56.677398 21673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:19:56.678403 21741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:56.680460 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 21673
I20260924 23:19:56.680660 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:19:56.680833 1043 external_mini_cluster.cc:1428] Setting key fdc283f3d1b8c48c7f0082a33cd9bc49
I20260924 23:19:56.680819 21741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe: Bootstrap starting.
I20260924 23:19:56.681460 21741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:56.681727 21741 log.cc:824] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:56.682360 21741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe: No bootstrap required, opened a new log
I20260924 23:19:56.684159 21741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb092e8c2b44b4399af7e8e7a887cbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35925 } }
I20260924 23:19:56.684319 21741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:56.684352 21741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb092e8c2b44b4399af7e8e7a887cbe, State: Initialized, Role: FOLLOWER
I20260924 23:19:56.684478 21741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb092e8c2b44b4399af7e8e7a887cbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35925 } }
I20260924 23:19:56.684540 21741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:19:56.684571 21741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:19:56.684623 21741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:56.685160 21741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb092e8c2b44b4399af7e8e7a887cbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35925 } }
I20260924 23:19:56.685271 21741 leader_election.cc:302] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9eb092e8c2b44b4399af7e8e7a887cbe; no voters:
I20260924 23:19:56.685424 21741 leader_election.cc:288] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:19:56.685470 21746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:56.685614 21746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [term 1 LEADER]: Becoming Leader. State: Replica: 9eb092e8c2b44b4399af7e8e7a887cbe, State: Running, Role: LEADER
I20260924 23:19:56.685699 21741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:19:56.685767 21746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb092e8c2b44b4399af7e8e7a887cbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35925 } }
I20260924 23:19:56.686107 21748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9eb092e8c2b44b4399af7e8e7a887cbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb092e8c2b44b4399af7e8e7a887cbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35925 } } }
I20260924 23:19:56.686290 21748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:56.686586 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:56.681421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51615 (local address 127.0.0.1:35925)
0924 23:19:56.681668 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:56.681677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:56.681702 (+ 25us) server_negotiation.cc:407] Connection header received
0924 23:19:56.682147 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:56.682154 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:56.682202 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:56.682325 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:56.682556 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:56.683504 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:56.684278 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:56.684815 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:56.684913 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:56.685656 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:56.685660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:56.685661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:56.685868 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:56.685912 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:56.685928 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:56.685994 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:56.686098 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 23:19:56.686206 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":72,"threads_started":1}
I20260924 23:19:56.686091 21746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eb092e8c2b44b4399af7e8e7a887cbe. Latest consensus state: current_term: 1 leader_uuid: "9eb092e8c2b44b4399af7e8e7a887cbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb092e8c2b44b4399af7e8e7a887cbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35925 } } }
I20260924 23:19:56.687057 21746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe [sys.catalog]: This master's current role is: LEADER
W20260924 23:19:56.687532 21762 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:19:56.687592 21762 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:19:56.687588 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:19:56.687733 21755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:19:56.687891 21755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:19:56.689342 21755 catalog_manager.cc:1347] Generated new cluster ID: 89c7282fb30c4e03bd0b559321b28dd1
I20260924 23:19:56.689391 21755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:19:56.692461 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:56.688996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54257 (local address 127.0.0.1:35925)
0924 23:19:56.689029 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:56.689031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:56.689137 (+ 106us) server_negotiation.cc:407] Connection header received
0924 23:19:56.689189 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:56.689190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:56.689200 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:56.689241 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:56.689428 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:56.690166 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:56.691174 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:56.691486 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:56.691521 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:56.692241 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:56.692243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:56.692243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:56.692247 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:56.692261 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:56.692264 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:56.692288 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:56.692372 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 23:19:56.692393 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 23:19:56.760476 21755 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:19:56.761237 21755 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:19:56.768783 21755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9eb092e8c2b44b4399af7e8e7a887cbe: Generated new TSK 0
I20260924 23:19:56.769001 21755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:19:56.780314 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 21673
2026-09-24T23:19:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:19:56 dist-test-slave-s1hw krb5kdc[21772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:19:56 dist-test-slave-s1hw krb5kdc[21772](info): set up 2 sockets
Sep 24 23:19:56 dist-test-slave-s1hw krb5kdc[21772](info): commencing operation
krb5kdc: starting...
W20260924 23:19:57.735436 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:19:57 dist-test-slave-s1hw krb5kdc[21772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:19:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:19:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:19:57.850497 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:57.929561 21785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:57.929775 21785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:57.929795 21785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:57.930922 21785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:19:57.930958 21785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:57.930970 21785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:57.930981 21785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:19:57.930992 21785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:19:57.932463 21785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:57.932677 21785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:57.932857 21785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:57.937963 21791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:57.937956 21793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:57.937956 21790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:57.938158 21785 server_base.cc:1034] running on GCE node
I20260924 23:19:57.938442 21785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:57.938728 21785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:57.939900 21785 hybrid_clock.cc:648] HybridClock initialized: now 1790291997939875 us; error 36 us; skew 500 ppm
Sep 24 23:19:57 dist-test-slave-s1hw krb5kdc[21772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:57.942394 21785 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:19:57.942935 21785 webserver.cc:460] Webserver started at http://127.0.0.1:33673/ using document root <none> and password file <none>
I20260924 23:19:57.943166 21785 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:57.943216 21785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:57.943343 21785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:57.944422 21785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "dba601900e834c878b2a52a7fc18f912"
format_stamp: "Formatted at 2026-09-24 23:19:57 on dist-test-slave-s1hw"
server_key: "d233f37e811e80ffc05ac9d62653e7e2"
server_key_iv: "b30f35461f5f79eef277d7a7e96761f5"
server_key_version: "encryptionkey@0"
I20260924 23:19:57.944756 21785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "dba601900e834c878b2a52a7fc18f912"
format_stamp: "Formatted at 2026-09-24 23:19:57 on dist-test-slave-s1hw"
server_key: "d233f37e811e80ffc05ac9d62653e7e2"
server_key_iv: "b30f35461f5f79eef277d7a7e96761f5"
server_key_version: "encryptionkey@0"
I20260924 23:19:57.946020 21785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:19:57.946790 21800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:57.946939 21785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:19:57.947023 21785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "dba601900e834c878b2a52a7fc18f912"
format_stamp: "Formatted at 2026-09-24 23:19:57 on dist-test-slave-s1hw"
server_key: "d233f37e811e80ffc05ac9d62653e7e2"
server_key_iv: "b30f35461f5f79eef277d7a7e96761f5"
server_key_version: "encryptionkey@0"
I20260924 23:19:57.947085 21785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:57.950416 21785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:57.952430 21785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:57.952551 21785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:57.956281 21785 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34877
I20260924 23:19:57.956310 21852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34877 every 8 connection(s)
I20260924 23:19:57.956612 21785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:19:57.957515 21853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:57.959836 21853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912: Bootstrap starting.
I20260924 23:19:57.960371 21853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:57.960613 21853 log.cc:824] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:57.961272 21853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912: No bootstrap required, opened a new log
I20260924 23:19:57.962771 21853 raft_consensus.cc:348] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba601900e834c878b2a52a7fc18f912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34877 } }
I20260924 23:19:57.962908 21853 raft_consensus.cc:374] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:57.962929 21853 raft_consensus.cc:729] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba601900e834c878b2a52a7fc18f912, State: Initialized, Role: FOLLOWER
I20260924 23:19:57.963060 21853 consensus_queue.cc:260] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba601900e834c878b2a52a7fc18f912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34877 } }
I20260924 23:19:57.963115 21853 raft_consensus.cc:388] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:19:57.963155 21853 raft_consensus.cc:482] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:19:57.963201 21853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:57.963742 21853 raft_consensus.cc:504] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba601900e834c878b2a52a7fc18f912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34877 } }
I20260924 23:19:57.963832 21853 leader_election.cc:302] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dba601900e834c878b2a52a7fc18f912; no voters:
I20260924 23:19:57.963979 21853 leader_election.cc:288] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:19:57.964008 21856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:57.964195 21856 raft_consensus.cc:686] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [term 1 LEADER]: Becoming Leader. State: Replica: dba601900e834c878b2a52a7fc18f912, State: Running, Role: LEADER
I20260924 23:19:57.964238 21853 sys_catalog.cc:564] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:19:57.964314 21856 consensus_queue.cc:237] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba601900e834c878b2a52a7fc18f912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34877 } }
I20260924 23:19:57.964677 21857 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dba601900e834c878b2a52a7fc18f912" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba601900e834c878b2a52a7fc18f912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34877 } } }
I20260924 23:19:57.964811 21857 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:57.964977 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 21785
I20260924 23:19:57.965094 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:19:57.965274 1043 external_mini_cluster.cc:1428] Setting key f819d954ab34aad5ea70e3fc0c79cdc8
I20260924 23:19:57.965293 21865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:19:57.965770 21865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:19:57.965831 21858 sys_catalog.cc:455] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dba601900e834c878b2a52a7fc18f912. Latest consensus state: current_term: 1 leader_uuid: "dba601900e834c878b2a52a7fc18f912" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dba601900e834c878b2a52a7fc18f912" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34877 } } }
I20260924 23:19:57.965981 21858 sys_catalog.cc:458] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912 [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:57.967229 21865 catalog_manager.cc:1347] Generated new cluster ID: b86055da6ff64eb7957c5ac515177b36
I20260924 23:19:57.967273 21865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:19:57.967690 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:57.965772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43305 (local address 127.0.0.1:34877)
0924 23:19:57.966077 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:57.966088 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:57.966325 (+ 237us) server_negotiation.cc:407] Connection header received
0924 23:19:57.966447 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:57.966453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:57.966477 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:57.966601 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:57.967262 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:57.967265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:57.967265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:57.967269 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:57.967303 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:57.967315 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:57.967386 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:57.967470 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 23:19:57.967512 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":61,"threads_started":1}
I20260924 23:19:57.968436 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:19:57.972527 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:57.969622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60095 (local address 127.0.0.1:34877)
0924 23:19:57.969663 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:57.969665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:57.971694 (+ 2029us) server_negotiation.cc:407] Connection header received
0924 23:19:57.971746 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:57.971747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:57.971753 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:57.971797 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:57.972329 (+ 532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:57.972330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:57.972331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:19:57.972335 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:57.972350 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:19:57.972354 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:57.972406 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:57.972450 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 23:19:57.972464 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:19:57.989506 21865 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:19:57.989748 21865 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:19:57.997534 21865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dba601900e834c878b2a52a7fc18f912: Generated new TSK 0
I20260924 23:19:57.997746 21865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:19:58.006248 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 21785
2026-09-24T23:19:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:19:58 dist-test-slave-s1hw krb5kdc[21884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:19:58 dist-test-slave-s1hw krb5kdc[21884](info): set up 2 sockets
Sep 24 23:19:58 dist-test-slave-s1hw krb5kdc[21884](info): commencing operation
krb5kdc: starting...
W20260924 23:19:58.963793 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:19:59 dist-test-slave-s1hw krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:19:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:19:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:19:59.082504 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36409
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:19:59.161062 21897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:19:59.161269 21897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:19:59.161298 21897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:19:59.162410 21897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:19:59.162451 21897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:19:59.162469 21897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:19:59.162483 21897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:19:59.162498 21897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:19:59.163992 21897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:19:59.164230 21897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:19:59.164433 21897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:19:59.168999 21902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:19:59.169190 21903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:59.169026 21897 server_base.cc:1034] running on GCE node
W20260924 23:19:59.169121 21905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:19:59.169490 21897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:19:59.169803 21897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:19:59.170953 21897 hybrid_clock.cc:648] HybridClock initialized: now 1790291999170936 us; error 29 us; skew 500 ppm
Sep 24 23:19:59 dist-test-slave-s1hw krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:19:59.173446 21897 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:19:59.173911 21897 webserver.cc:460] Webserver started at http://127.0.0.1:34769/ using document root <none> and password file <none>
I20260924 23:19:59.174110 21897 fs_manager.cc:359] Metadata directory not provided
I20260924 23:19:59.174156 21897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:19:59.174276 21897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:19:59.175297 21897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "0bee3a513bc04b04ae51d6a26c41e169"
format_stamp: "Formatted at 2026-09-24 23:19:59 on dist-test-slave-s1hw"
server_key: "1b0e8dc44e62cd99ece29d6201d15170"
server_key_iv: "c8e5ca4b11ad9826828becd129ee8c12"
server_key_version: "encryptionkey@0"
I20260924 23:19:59.175685 21897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "0bee3a513bc04b04ae51d6a26c41e169"
format_stamp: "Formatted at 2026-09-24 23:19:59 on dist-test-slave-s1hw"
server_key: "1b0e8dc44e62cd99ece29d6201d15170"
server_key_iv: "c8e5ca4b11ad9826828becd129ee8c12"
server_key_version: "encryptionkey@0"
I20260924 23:19:59.176973 21897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:59.177742 21912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:19:59.177909 21897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:19:59.177994 21897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "0bee3a513bc04b04ae51d6a26c41e169"
format_stamp: "Formatted at 2026-09-24 23:19:59 on dist-test-slave-s1hw"
server_key: "1b0e8dc44e62cd99ece29d6201d15170"
server_key_iv: "c8e5ca4b11ad9826828becd129ee8c12"
server_key_version: "encryptionkey@0"
I20260924 23:19:59.178056 21897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:19:59.186785 21897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:19:59.189106 21897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:19:59.189248 21897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:19:59.193007 21897 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36409
I20260924 23:19:59.193042 21964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36409 every 8 connection(s)
I20260924 23:19:59.193372 21897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:19:59.194351 21965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:19:59.196650 21965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169: Bootstrap starting.
I20260924 23:19:59.196691 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 21897
I20260924 23:19:59.196803 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:19:59.196961 1043 external_mini_cluster.cc:1428] Setting key 3124a7ee6448e7b3c6c8b7482bfb7b5a
I20260924 23:19:59.197216 21965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169: Neither blocks nor log segments found. Creating new log.
I20260924 23:19:59.197517 21965 log.cc:824] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169: Log is configured to *not* fsync() on all Append() calls
I20260924 23:19:59.198151 21965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169: No bootstrap required, opened a new log
I20260924 23:19:59.199944 21965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bee3a513bc04b04ae51d6a26c41e169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36409 } }
I20260924 23:19:59.200100 21965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:19:59.200126 21965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bee3a513bc04b04ae51d6a26c41e169, State: Initialized, Role: FOLLOWER
I20260924 23:19:59.200253 21965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bee3a513bc04b04ae51d6a26c41e169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36409 } }
I20260924 23:19:59.200330 21965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:19:59.200368 21965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:19:59.200415 21965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:19:59.200938 21965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bee3a513bc04b04ae51d6a26c41e169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36409 } }
I20260924 23:19:59.201035 21965 leader_election.cc:302] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bee3a513bc04b04ae51d6a26c41e169; no voters:
I20260924 23:19:59.201187 21965 leader_election.cc:288] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:19:59.201227 21970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:19:59.201370 21970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [term 1 LEADER]: Becoming Leader. State: Replica: 0bee3a513bc04b04ae51d6a26c41e169, State: Running, Role: LEADER
I20260924 23:19:59.201449 21965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:19:59.201476 21970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bee3a513bc04b04ae51d6a26c41e169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36409 } }
I20260924 23:19:59.201781 21970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bee3a513bc04b04ae51d6a26c41e169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bee3a513bc04b04ae51d6a26c41e169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36409 } } }
I20260924 23:19:59.201901 21970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:59.202077 21970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bee3a513bc04b04ae51d6a26c41e169. Latest consensus state: current_term: 1 leader_uuid: "0bee3a513bc04b04ae51d6a26c41e169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bee3a513bc04b04ae51d6a26c41e169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36409 } } }
I20260924 23:19:59.202191 21970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169 [sys.catalog]: This master's current role is: LEADER
I20260924 23:19:59.202338 21977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:19:59.202831 21977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 23:19:59 dist-test-slave-s1hw krb5kdc[21884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 23:19:59.204423 21977 catalog_manager.cc:1347] Generated new cluster ID: ea497813be6545dd9a7d964a46daa3c2
I20260924 23:19:59.205050 21977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:19:59.206990 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:59.197441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:36075 (local address 127.0.0.1:36409)
0924 23:19:59.197613 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:59.197624 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:59.197657 (+ 33us) server_negotiation.cc:407] Connection header received
0924 23:19:59.198084 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:59.198093 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:59.198143 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:59.198285 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:59.198880 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:59.199857 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:59.200810 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:59.201286 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:59.201390 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:59.203986 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:59.203990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:59.203990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:59.204004 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:59.205585 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:59.205939 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:59.205942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:59.205943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:59.206023 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:59.206229 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:59.206230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:59.206230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:59.206458 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:59.206622 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:59.206746 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 23:19:59.206839 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260924 23:19:59.207988 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:19:59.211561 21977 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:19:59.212348 21977 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:19:59.217787 21977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bee3a513bc04b04ae51d6a26c41e169: Generated new TSK 0
I20260924 23:19:59.218006 21977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:19:59.221357 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:19:59.211688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:44339 (local address 127.0.0.1:36409)
0924 23:19:59.211731 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:19:59.211733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:19:59.215587 (+ 3854us) server_negotiation.cc:407] Connection header received
0924 23:19:59.215662 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:19:59.215663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:19:59.215674 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:19:59.215723 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:19:59.216250 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:59.216809 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:59.217591 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:19:59.217811 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:19:59.217866 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:19:59.219043 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:19:59.219045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:19:59.219045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:19:59.219052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:19:59.220319 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:59.220668 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:59.220669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:59.220669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:59.220715 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:19:59.220917 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:19:59.220918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:19:59.220918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:19:59.221013 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:19:59.221113 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 23:19:59.221218 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 23:19:59.221275 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:19:59.223153 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 21897
W20260924 23:19:59.226683 21882 connection.cc:535] client connection to 127.0.0.1:36409 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36409): Connection reset by peer (error 104)
2026-09-24T23:19:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:19:59 dist-test-slave-s1hw krb5kdc[21996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:19:59 dist-test-slave-s1hw krb5kdc[21996](info): set up 2 sockets
Sep 24 23:19:59 dist-test-slave-s1hw krb5kdc[21996](info): commencing operation
krb5kdc: starting...
W20260924 23:20:00.179209 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:20:00 dist-test-slave-s1hw krb5kdc[21996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:20:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:20:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:20:00.298164 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:20:00.379032 22009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:00.379276 22009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:00.379300 22009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:20:00.380535 22009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:00.380586 22009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:00.380600 22009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:00.380618 22009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:00.380630 22009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:20:00.382112 22009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:00.382349 22009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:00.382578 22009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:00.387657 22017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:00.387682 22009 server_base.cc:1034] running on GCE node
W20260924 23:20:00.387665 22014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:00.387661 22015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:00.388214 22009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:00.388526 22009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:00.389703 22009 hybrid_clock.cc:648] HybridClock initialized: now 1790292000389674 us; error 40 us; skew 500 ppm
Sep 24 23:20:00 dist-test-slave-s1hw krb5kdc[21996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:00.392689 22009 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:20:00.393247 22009 webserver.cc:460] Webserver started at http://127.0.0.1:45997/ using document root <none> and password file <none>
I20260924 23:20:00.393474 22009 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:00.393539 22009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:00.393678 22009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:00.394840 22009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "8e2045f0586340e0a701f6ea38b3eaa4"
format_stamp: "Formatted at 2026-09-24 23:20:00 on dist-test-slave-s1hw"
server_key: "e22328f5ca1ac856c083e17f36a16bcc"
server_key_iv: "65994eb1f0c7159f853ce6a4a3c8de3f"
server_key_version: "encryptionkey@0"
I20260924 23:20:00.395213 22009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "8e2045f0586340e0a701f6ea38b3eaa4"
format_stamp: "Formatted at 2026-09-24 23:20:00 on dist-test-slave-s1hw"
server_key: "e22328f5ca1ac856c083e17f36a16bcc"
server_key_iv: "65994eb1f0c7159f853ce6a4a3c8de3f"
server_key_version: "encryptionkey@0"
I20260924 23:20:00.396872 22009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:20:00.397814 22024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:00.398066 22009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:20:00.398169 22009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "8e2045f0586340e0a701f6ea38b3eaa4"
format_stamp: "Formatted at 2026-09-24 23:20:00 on dist-test-slave-s1hw"
server_key: "e22328f5ca1ac856c083e17f36a16bcc"
server_key_iv: "65994eb1f0c7159f853ce6a4a3c8de3f"
server_key_version: "encryptionkey@0"
I20260924 23:20:00.398252 22009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:00.422153 22009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:00.424544 22009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:00.424701 22009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:00.428746 22009 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37853
I20260924 23:20:00.428783 22076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37853 every 8 connection(s)
I20260924 23:20:00.429131 22009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:20:00.430163 22077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:00.432614 22077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4: Bootstrap starting.
I20260924 23:20:00.433243 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 22009
I20260924 23:20:00.433272 22077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:00.433367 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:20:00.433607 1043 external_mini_cluster.cc:1428] Setting key c80902dfe030e27ceaa9cb551c8b41e6
I20260924 23:20:00.433617 22077 log.cc:824] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:00.434423 22077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4: No bootstrap required, opened a new log
I20260924 23:20:00.436318 22077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2045f0586340e0a701f6ea38b3eaa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37853 } }
I20260924 23:20:00.436510 22077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:20:00.436563 22077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e2045f0586340e0a701f6ea38b3eaa4, State: Initialized, Role: FOLLOWER
I20260924 23:20:00.436691 22077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2045f0586340e0a701f6ea38b3eaa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37853 } }
I20260924 23:20:00.436784 22077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:20:00.436826 22077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:20:00.436882 22077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:20:00.437691 22077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2045f0586340e0a701f6ea38b3eaa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37853 } }
I20260924 23:20:00.437813 22077 leader_election.cc:302] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e2045f0586340e0a701f6ea38b3eaa4; no voters:
I20260924 23:20:00.438007 22077 leader_election.cc:288] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:20:00.438062 22082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:20:00.438238 22082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [term 1 LEADER]: Becoming Leader. State: Replica: 8e2045f0586340e0a701f6ea38b3eaa4, State: Running, Role: LEADER
I20260924 23:20:00.438355 22077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:00.438374 22082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2045f0586340e0a701f6ea38b3eaa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37853 } }
I20260924 23:20:00.439037 22083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e2045f0586340e0a701f6ea38b3eaa4. Latest consensus state: current_term: 1 leader_uuid: "8e2045f0586340e0a701f6ea38b3eaa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2045f0586340e0a701f6ea38b3eaa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37853 } } }
I20260924 23:20:00.439147 22083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:00.439639 22084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e2045f0586340e0a701f6ea38b3eaa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2045f0586340e0a701f6ea38b3eaa4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37853 } } }
I20260924 23:20:00.439795 22084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4 [sys.catalog]: This master's current role is: LEADER
W20260924 23:20:00.440271 22098 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:20:00.440328 22098 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:20:00.440368 22095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:20:00.440519 22095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:20:00.442198 22095 catalog_manager.cc:1347] Generated new cluster ID: 8746840eda4b42ea8d4eec81d9888810
I20260924 23:20:00.442236 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:00.434150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:50041 (local address 127.0.0.1:37853)
0924 23:20:00.434322 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:00.434332 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:00.435576 (+ 1244us) server_negotiation.cc:407] Connection header received
0924 23:20:00.436271 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:00.436281 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:00.436354 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:00.436543 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:00.436756 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:00.438419 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:00.439570 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:00.440203 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:00.440342 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:00.441340 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:00.441344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:00.441345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:20:00.441579 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:00.441622 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:20:00.441639 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:00.441738 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:00.441959 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 23:20:00.441991 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260924 23:20:00.442258 22095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:20:00.443289 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:20:00.448264 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:00.444670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:36085 (local address 127.0.0.1:37853)
0924 23:20:00.444724 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:00.444726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:00.444836 (+ 110us) server_negotiation.cc:407] Connection header received
0924 23:20:00.444905 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:00.444906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:00.444927 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:00.444982 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:00.445112 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:00.445909 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:00.446881 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:00.447198 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:00.447277 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:00.448013 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:00.448015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:00.448015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:20:00.448020 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:00.448039 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:20:00.448043 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:00.448106 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:00.448158 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 23:20:00.448174 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 23:20:00.471004 22095 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:20:00.471828 22095 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:20:00.476198 22095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e2045f0586340e0a701f6ea38b3eaa4: Generated new TSK 0
I20260924 23:20:00.476423 22095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:20:00.478492 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 22009
W20260924 23:20:00.482827 21994 connection.cc:535] client connection to 127.0.0.1:37853 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37853): Connection reset by peer (error 104)
2026-09-24T23:20:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1256 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:20:00 dist-test-slave-s1hw krb5kdc[22108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:20:00 dist-test-slave-s1hw krb5kdc[22108](info): set up 2 sockets
Sep 24 23:20:00 dist-test-slave-s1hw krb5kdc[22108](info): commencing operation
krb5kdc: starting...
W20260924 23:20:01.443339 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:20:01 dist-test-slave-s1hw krb5kdc[22108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:20:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:20:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:20:01.564558 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:20:01.650957 22121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:01.651239 22121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:01.651276 22121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:20:01.653013 22121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:01.653088 22121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:01.653108 22121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:01.653126 22121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:01.653141 22121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:20:01.655045 22121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:01.655280 22121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:01.655512 22121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:01.660324 22127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:01.660317 22126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:01.660317 22129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:01.660501 22121 server_base.cc:1034] running on GCE node
I20260924 23:20:01.660846 22121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:01.661115 22121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:01.662274 22121 hybrid_clock.cc:648] HybridClock initialized: now 1790292001662254 us; error 30 us; skew 500 ppm
Sep 24 23:20:01 dist-test-slave-s1hw krb5kdc[22108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:01.664966 22121 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:20:01.665438 22121 webserver.cc:460] Webserver started at http://127.0.0.1:42749/ using document root <none> and password file <none>
I20260924 23:20:01.665679 22121 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:01.665729 22121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:01.665859 22121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:01.666936 22121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "229074e527b0482f857c9f8f7fc971cc"
format_stamp: "Formatted at 2026-09-24 23:20:01 on dist-test-slave-s1hw"
server_key: "432d4b4b8b36388e9127abe40b2d2ced"
server_key_iv: "4567070c23505dfbf1442ba235a20d77"
server_key_version: "encryptionkey@0"
I20260924 23:20:01.667263 22121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "229074e527b0482f857c9f8f7fc971cc"
format_stamp: "Formatted at 2026-09-24 23:20:01 on dist-test-slave-s1hw"
server_key: "432d4b4b8b36388e9127abe40b2d2ced"
server_key_iv: "4567070c23505dfbf1442ba235a20d77"
server_key_version: "encryptionkey@0"
I20260924 23:20:01.668608 22121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:20:01.669425 22136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:01.669669 22121 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:20:01.669775 22121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "229074e527b0482f857c9f8f7fc971cc"
format_stamp: "Formatted at 2026-09-24 23:20:01 on dist-test-slave-s1hw"
server_key: "432d4b4b8b36388e9127abe40b2d2ced"
server_key_iv: "4567070c23505dfbf1442ba235a20d77"
server_key_version: "encryptionkey@0"
I20260924 23:20:01.669839 22121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:01.673257 22121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:01.675318 22121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:01.675477 22121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:01.679402 22121 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45877
I20260924 23:20:01.679420 22188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45877 every 8 connection(s)
I20260924 23:20:01.679788 22121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:20:01.680745 22189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:01.683019 22189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc: Bootstrap starting.
I20260924 23:20:01.683621 22189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:01.683868 22189 log.cc:824] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:01.684541 22189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc: No bootstrap required, opened a new log
I20260924 23:20:01.686117 22189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229074e527b0482f857c9f8f7fc971cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45877 } }
I20260924 23:20:01.686307 22189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:20:01.686345 22189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229074e527b0482f857c9f8f7fc971cc, State: Initialized, Role: FOLLOWER
I20260924 23:20:01.686486 22189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229074e527b0482f857c9f8f7fc971cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45877 } }
I20260924 23:20:01.686550 22189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:20:01.686579 22189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:20:01.686626 22189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:20:01.687182 22189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229074e527b0482f857c9f8f7fc971cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45877 } }
I20260924 23:20:01.687290 22189 leader_election.cc:302] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 229074e527b0482f857c9f8f7fc971cc; no voters:
I20260924 23:20:01.687511 22189 leader_election.cc:288] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:20:01.687556 22192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:20:01.687665 22192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [term 1 LEADER]: Becoming Leader. State: Replica: 229074e527b0482f857c9f8f7fc971cc, State: Running, Role: LEADER
I20260924 23:20:01.687785 22189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:01.687748 22192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229074e527b0482f857c9f8f7fc971cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45877 } }
I20260924 23:20:01.688194 22193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "229074e527b0482f857c9f8f7fc971cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229074e527b0482f857c9f8f7fc971cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45877 } } }
I20260924 23:20:01.688278 22193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:01.688324 22194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 229074e527b0482f857c9f8f7fc971cc. Latest consensus state: current_term: 1 leader_uuid: "229074e527b0482f857c9f8f7fc971cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229074e527b0482f857c9f8f7fc971cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45877 } } }
I20260924 23:20:01.688433 22194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:01.688489 22199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:20:01.689018 22199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:20:01.689431 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 22121
I20260924 23:20:01.689539 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:20:01.689738 1043 external_mini_cluster.cc:1428] Setting key 69076161a11c12a4bb0d81ce210706c7
I20260924 23:20:01.690872 22199 catalog_manager.cc:1347] Generated new cluster ID: 51bc412beee04ec08c8e0ac8babf2b4a
I20260924 23:20:01.690927 22199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:20:01.696336 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:01.690181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:45335 (local address 127.0.0.1:45877)
0924 23:20:01.690356 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:01.690365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:01.690399 (+ 34us) server_negotiation.cc:407] Connection header received
0924 23:20:01.690588 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:01.690596 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:01.690628 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:01.690774 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:01.695809 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:01.695813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:01.695813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:20:01.695817 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:01.695849 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:20:01.695861 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:01.695952 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:01.696046 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 23:20:01.696137 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260924 23:20:01.697499 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:20:01.702557 22199 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:20:01.702782 22199 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:20:01.703876 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:01.702387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:36479 (local address 127.0.0.1:45877)
0924 23:20:01.702425 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:01.702427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:01.702603 (+ 176us) server_negotiation.cc:407] Connection header received
0924 23:20:01.702710 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:01.702711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:01.702718 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:01.702751 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:01.703623 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:01.703625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:01.703625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:20:01.703630 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:01.703650 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:20:01.703655 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:01.703724 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:01.703783 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 23:20:01.703798 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:20:01.707424 22199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 229074e527b0482f857c9f8f7fc971cc: Generated new TSK 0
I20260924 23:20:01.707628 22199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:20:01.711256 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 22121
2026-09-24T23:20:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1232 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:20:01 dist-test-slave-s1hw krb5kdc[22220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:20:01 dist-test-slave-s1hw krb5kdc[22220](info): set up 2 sockets
Sep 24 23:20:01 dist-test-slave-s1hw krb5kdc[22220](info): commencing operation
krb5kdc: starting...
W20260924 23:20:02.671388 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:20:02 dist-test-slave-s1hw krb5kdc[22220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:20:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:20:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:20:02.791157 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:20:02.873528 22233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:02.873749 22233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:02.873782 22233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:20:02.874943 22233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:02.874986 22233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:02.875003 22233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:02.875020 22233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:02.875033 22233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:20:02.876569 22233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39235
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:02.876834 22233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:02.877041 22233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:02.881778 22238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:02.881795 22239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:02.881846 22233 server_base.cc:1034] running on GCE node
W20260924 23:20:02.881964 22241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:02.882277 22233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:02.882537 22233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:02.883672 22233 hybrid_clock.cc:648] HybridClock initialized: now 1790292002883656 us; error 35 us; skew 500 ppm
Sep 24 23:20:02 dist-test-slave-s1hw krb5kdc[22220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:02.886018 22233 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:20:02.886456 22233 webserver.cc:460] Webserver started at http://127.0.0.1:42233/ using document root <none> and password file <none>
I20260924 23:20:02.886657 22233 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:02.886696 22233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:02.886787 22233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:02.887848 22233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "2c6af364819e4900a733c83cd1b1e421"
format_stamp: "Formatted at 2026-09-24 23:20:02 on dist-test-slave-s1hw"
server_key: "cd8d416ac912fab6219cbbd599a46039"
server_key_iv: "45799d82117e23e2d6e969d533a32ce1"
server_key_version: "encryptionkey@0"
I20260924 23:20:02.888139 22233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "2c6af364819e4900a733c83cd1b1e421"
format_stamp: "Formatted at 2026-09-24 23:20:02 on dist-test-slave-s1hw"
server_key: "cd8d416ac912fab6219cbbd599a46039"
server_key_iv: "45799d82117e23e2d6e969d533a32ce1"
server_key_version: "encryptionkey@0"
I20260924 23:20:02.889320 22233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:20:02.889979 22248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:02.890134 22233 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:20:02.890201 22233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "2c6af364819e4900a733c83cd1b1e421"
format_stamp: "Formatted at 2026-09-24 23:20:02 on dist-test-slave-s1hw"
server_key: "cd8d416ac912fab6219cbbd599a46039"
server_key_iv: "45799d82117e23e2d6e969d533a32ce1"
server_key_version: "encryptionkey@0"
I20260924 23:20:02.890288 22233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:02.917853 22233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:02.920243 22233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:02.920368 22233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:02.924114 22233 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39235
I20260924 23:20:02.924160 22300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39235 every 8 connection(s)
I20260924 23:20:02.924503 22233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:20:02.925318 22301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:02.925946 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 22233
I20260924 23:20:02.926036 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:20:02.926210 1043 external_mini_cluster.cc:1428] Setting key e7a76b40e338d09c0bb691ffb38e4a13
I20260924 23:20:02.928282 22301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421: Bootstrap starting.
I20260924 23:20:02.928948 22301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:02.929329 22301 log.cc:824] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:02.930032 22301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421: No bootstrap required, opened a new log
I20260924 23:20:02.932173 22301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6af364819e4900a733c83cd1b1e421" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39235 } }
I20260924 23:20:02.932319 22301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:20:02.932343 22301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6af364819e4900a733c83cd1b1e421, State: Initialized, Role: FOLLOWER
I20260924 23:20:02.932456 22301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6af364819e4900a733c83cd1b1e421" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39235 } }
I20260924 23:20:02.932521 22301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:20:02.932548 22301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:20:02.932592 22301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 23:20:02 dist-test-slave-s1hw krb5kdc[22220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 23:20:02.933264 22301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6af364819e4900a733c83cd1b1e421" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39235 } }
I20260924 23:20:02.933368 22301 leader_election.cc:302] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c6af364819e4900a733c83cd1b1e421; no voters:
I20260924 23:20:02.933564 22301 leader_election.cc:288] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:20:02.933605 22306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:20:02.933758 22306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [term 1 LEADER]: Becoming Leader. State: Replica: 2c6af364819e4900a733c83cd1b1e421, State: Running, Role: LEADER
I20260924 23:20:02.933820 22301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:02.933889 22306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6af364819e4900a733c83cd1b1e421" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39235 } }
I20260924 23:20:02.934283 22306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c6af364819e4900a733c83cd1b1e421. Latest consensus state: current_term: 1 leader_uuid: "2c6af364819e4900a733c83cd1b1e421" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6af364819e4900a733c83cd1b1e421" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39235 } } }
I20260924 23:20:02.934319 22308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c6af364819e4900a733c83cd1b1e421" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6af364819e4900a733c83cd1b1e421" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39235 } } }
I20260924 23:20:02.934504 22308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:02.934422 22306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:02.934890 22315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:20:02.935428 22315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:20:02.937250 22315 catalog_manager.cc:1347] Generated new cluster ID: cdfcc549a03249c3a49ab1d4c791ecb4
I20260924 23:20:02.937347 22315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:20:02.938313 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:02.926755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:57341 (local address 127.0.0.1:39235)
0924 23:20:02.926936 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:02.926945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:02.926978 (+ 33us) server_negotiation.cc:407] Connection header received
0924 23:20:02.927518 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:02.927527 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:02.927601 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:02.927776 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:02.928372 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:02.929696 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:02.930533 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:02.931159 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:02.931275 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:02.933747 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:02.933751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:02.933751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:02.933763 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:02.936485 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:02.936965 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:02.936968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:02.936969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:02.937073 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:02.937347 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:02.937348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:02.937348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:02.937686 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:02.937844 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:02.937975 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 23:20:02.938084 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260924 23:20:02.939266 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:20:02.947919 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:02.940848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:56213 (local address 127.0.0.1:39235)
0924 23:20:02.940879 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:02.940881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:02.940913 (+ 32us) server_negotiation.cc:407] Connection header received
0924 23:20:02.940993 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:02.940993 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:02.941002 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:02.941049 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:02.941671 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:02.942387 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:02.943446 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:02.943759 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:02.943812 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:02.945215 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:02.945217 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:02.945218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:02.945225 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:02.946675 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:02.947072 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:02.947073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:02.947073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:02.947119 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:02.947299 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:02.947300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:02.947300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:02.947453 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:02.947528 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:02.947755 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 23:20:02.947819 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 23:20:02.972158 22315 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:20:02.972739 22315 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:20:02.982581 22315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c6af364819e4900a733c83cd1b1e421: Generated new TSK 0
I20260924 23:20:02.982779 22315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:20:02.987731 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 22233
W20260924 23:20:02.991643 22218 connection.cc:535] client connection to 127.0.0.1:39235 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39235): Connection reset by peer (error 104)
2026-09-24T23:20:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1275 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:20:03 dist-test-slave-s1hw krb5kdc[22332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:20:03 dist-test-slave-s1hw krb5kdc[22332](info): set up 2 sockets
Sep 24 23:20:03 dist-test-slave-s1hw krb5kdc[22332](info): commencing operation
krb5kdc: starting...
W20260924 23:20:03.943238 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:20:04 dist-test-slave-s1hw krb5kdc[22332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:20:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:20:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:20:04.065922 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33627
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:20:04.146270 22345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:04.146492 22345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:04.146524 22345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:20:04.147799 22345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:04.147850 22345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:04.147863 22345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:04.147874 22345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:04.147886 22345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:20:04.149425 22345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33627
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:04.149678 22345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:04.149895 22345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:04.154955 22351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:04.154979 22350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:04.154964 22353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:04.154994 22345 server_base.cc:1034] running on GCE node
I20260924 23:20:04.155522 22345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:04.155797 22345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:04.156949 22345 hybrid_clock.cc:648] HybridClock initialized: now 1790292004156929 us; error 36 us; skew 500 ppm
Sep 24 23:20:04 dist-test-slave-s1hw krb5kdc[22332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:04.159584 22345 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:20:04.160054 22345 webserver.cc:460] Webserver started at http://127.0.0.1:34867/ using document root <none> and password file <none>
I20260924 23:20:04.160266 22345 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:04.160319 22345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:04.160414 22345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:04.161466 22345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "fb67dfb531504462afc79f809ffc69f2"
format_stamp: "Formatted at 2026-09-24 23:20:04 on dist-test-slave-s1hw"
server_key: "5fecade4d13de4103f2b01bae2b3807d"
server_key_iv: "813e1d51730dbf9d6966fc987c0feb1c"
server_key_version: "encryptionkey@0"
I20260924 23:20:04.161777 22345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "fb67dfb531504462afc79f809ffc69f2"
format_stamp: "Formatted at 2026-09-24 23:20:04 on dist-test-slave-s1hw"
server_key: "5fecade4d13de4103f2b01bae2b3807d"
server_key_iv: "813e1d51730dbf9d6966fc987c0feb1c"
server_key_version: "encryptionkey@0"
I20260924 23:20:04.163034 22345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:20:04.163823 22360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:04.163978 22345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:04.164065 22345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "fb67dfb531504462afc79f809ffc69f2"
format_stamp: "Formatted at 2026-09-24 23:20:04 on dist-test-slave-s1hw"
server_key: "5fecade4d13de4103f2b01bae2b3807d"
server_key_iv: "813e1d51730dbf9d6966fc987c0feb1c"
server_key_version: "encryptionkey@0"
I20260924 23:20:04.164137 22345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:04.199440 22345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:04.201869 22345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:04.202033 22345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:04.206072 22345 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33627
I20260924 23:20:04.206116 22412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33627 every 8 connection(s)
I20260924 23:20:04.206435 22345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:20:04.207283 22413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:04.209591 22413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2: Bootstrap starting.
I20260924 23:20:04.210137 22413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:04.210387 22413 log.cc:824] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:04.210912 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 22345
I20260924 23:20:04.211021 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:20:04.211174 1043 external_mini_cluster.cc:1428] Setting key 75c687cefb17ce3a15012b90c899aa57
I20260924 23:20:04.211200 22413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2: No bootstrap required, opened a new log
I20260924 23:20:04.214149 22413 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb67dfb531504462afc79f809ffc69f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33627 } }
I20260924 23:20:04.214331 22413 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:20:04.214362 22413 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb67dfb531504462afc79f809ffc69f2, State: Initialized, Role: FOLLOWER
I20260924 23:20:04.214494 22413 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb67dfb531504462afc79f809ffc69f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33627 } }
I20260924 23:20:04.214578 22413 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:20:04.214603 22413 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:20:04.214653 22413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:20:04.215250 22413 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb67dfb531504462afc79f809ffc69f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33627 } }
I20260924 23:20:04.215356 22413 leader_election.cc:302] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb67dfb531504462afc79f809ffc69f2; no voters:
I20260924 23:20:04.215533 22413 leader_election.cc:288] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:20:04.215574 22418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:20:04.215732 22418 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [term 1 LEADER]: Becoming Leader. State: Replica: fb67dfb531504462afc79f809ffc69f2, State: Running, Role: LEADER
I20260924 23:20:04.215781 22413 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:04.215916 22418 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb67dfb531504462afc79f809ffc69f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33627 } }
I20260924 23:20:04.216502 22419 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb67dfb531504462afc79f809ffc69f2. Latest consensus state: current_term: 1 leader_uuid: "fb67dfb531504462afc79f809ffc69f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb67dfb531504462afc79f809ffc69f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33627 } } }
I20260924 23:20:04.216578 22419 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [sys.catalog]: This master's current role is: LEADER
W20260924 23:20:04.217542 22434 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:20:04.217644 22434 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:20:04.217686 22420 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb67dfb531504462afc79f809ffc69f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb67dfb531504462afc79f809ffc69f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33627 } } }
I20260924 23:20:04.217782 22420 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:04.217872 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:04.211736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:41241 (local address 127.0.0.1:33627)
0924 23:20:04.211931 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:04.211941 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:04.211982 (+ 41us) server_negotiation.cc:407] Connection header received
0924 23:20:04.212653 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:04.212665 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:04.212735 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:04.212892 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:04.213044 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:04.214410 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:04.215271 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:04.215958 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:04.216094 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:04.217006 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:04.217010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:04.217010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:20:04.217215 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:04.217257 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:20:04.217274 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:04.217352 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:04.217588 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 23:20:04.217618 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260924 23:20:04.217949 22427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:20:04.218123 22427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:20:04.218961 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:20:04.219678 22427 catalog_manager.cc:1347] Generated new cluster ID: d9d10dfe7e2b4226b6ae701083e06aed
I20260924 23:20:04.219727 22427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:20:04.224187 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:04.220483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:49079 (local address 127.0.0.1:33627)
0924 23:20:04.220524 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:04.220525 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:04.220653 (+ 128us) server_negotiation.cc:407] Connection header received
0924 23:20:04.220703 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:04.220704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:04.220715 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:04.220760 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:04.220907 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:04.221643 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:04.222743 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:04.223076 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:04.223151 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:04.223932 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:04.223933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:04.223933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:20:04.223939 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:04.223955 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:20:04.223959 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:04.224011 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:04.224092 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 23:20:04.224103 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:20:04.248641 22427 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:20:04.249191 22427 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:20:04.260433 22427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb67dfb531504462afc79f809ffc69f2: Generated new TSK 0
I20260924 23:20:04.260643 22427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:20:04.263477 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 22345
W20260924 23:20:04.267318 22330 connection.cc:535] client connection to 127.0.0.1:33627 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33627): Connection reset by peer (error 104)
2026-09-24T23:20:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1275 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:20:04 dist-test-slave-s1hw krb5kdc[22444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:20:04 dist-test-slave-s1hw krb5kdc[22444](info): set up 2 sockets
Sep 24 23:20:04 dist-test-slave-s1hw krb5kdc[22444](info): commencing operation
krb5kdc: starting...
W20260924 23:20:05.219260 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:20:05 dist-test-slave-s1hw krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:20:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:20:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:20:05.333630 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:20:05.413038 22457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:05.413268 22457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:05.413300 22457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:20:05.414507 22457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:05.414558 22457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:05.414577 22457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:05.414593 22457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:05.414608 22457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:20:05.416185 22457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35575
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:05.416443 22457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:05.416668 22457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:05.421532 22465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:05.421545 22457 server_base.cc:1034] running on GCE node
W20260924 23:20:05.421516 22463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:05.421516 22462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:05.422015 22457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:05.422333 22457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:05.423465 22457 hybrid_clock.cc:648] HybridClock initialized: now 1790292005423446 us; error 31 us; skew 500 ppm
Sep 24 23:20:05 dist-test-slave-s1hw krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:05.429220 22457 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:20:05.429738 22457 webserver.cc:460] Webserver started at http://127.0.0.1:42695/ using document root <none> and password file <none>
I20260924 23:20:05.429955 22457 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:05.430004 22457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:05.430137 22457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:05.431281 22457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "c2d0649bcab0441497de832d0b77d44b"
format_stamp: "Formatted at 2026-09-24 23:20:05 on dist-test-slave-s1hw"
server_key: "02ebbe04fa2c768c0c2b447195f42939"
server_key_iv: "8231b679ca0bf5b6131f89d334eaa651"
server_key_version: "encryptionkey@0"
I20260924 23:20:05.431674 22457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "c2d0649bcab0441497de832d0b77d44b"
format_stamp: "Formatted at 2026-09-24 23:20:05 on dist-test-slave-s1hw"
server_key: "02ebbe04fa2c768c0c2b447195f42939"
server_key_iv: "8231b679ca0bf5b6131f89d334eaa651"
server_key_version: "encryptionkey@0"
I20260924 23:20:05.432979 22457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:20:05.433741 22472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:05.433897 22457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:20:05.433986 22457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "c2d0649bcab0441497de832d0b77d44b"
format_stamp: "Formatted at 2026-09-24 23:20:05 on dist-test-slave-s1hw"
server_key: "02ebbe04fa2c768c0c2b447195f42939"
server_key_iv: "8231b679ca0bf5b6131f89d334eaa651"
server_key_version: "encryptionkey@0"
I20260924 23:20:05.434053 22457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:05.437459 22457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:05.439601 22457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:05.439730 22457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:05.443679 22457 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35575
I20260924 23:20:05.443722 22524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35575 every 8 connection(s)
I20260924 23:20:05.444057 22457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:20:05.445065 22525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:05.447479 22525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b: Bootstrap starting.
I20260924 23:20:05.448043 22525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:05.448285 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 22457
I20260924 23:20:05.448307 22525 log.cc:824] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:05.448382 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:20:05.448618 1043 external_mini_cluster.cc:1428] Setting key 28c1942ed0065ca626016e5bbfde0313
I20260924 23:20:05.449088 22525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b: No bootstrap required, opened a new log
I20260924 23:20:05.451210 22525 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d0649bcab0441497de832d0b77d44b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35575 } }
I20260924 23:20:05.451417 22525 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:20:05.451488 22525 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d0649bcab0441497de832d0b77d44b, State: Initialized, Role: FOLLOWER
I20260924 23:20:05.451629 22525 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d0649bcab0441497de832d0b77d44b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35575 } }
I20260924 23:20:05.451743 22525 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:20:05.451774 22525 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:20:05.451820 22525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:20:05.451948 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:05.449129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:33309 (local address 127.0.0.1:35575)
0924 23:20:05.449333 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:05.449343 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:05.449376 (+ 33us) server_negotiation.cc:407] Connection header received
0924 23:20:05.450337 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:05.450348 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:05.450386 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:05.450543 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:05.451196 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:05.451200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:05.451201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:20:05.451205 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:05.451249 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:20:05.451264 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:05.451338 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:05.451496 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 23:20:05.451542 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260924 23:20:05.452428 22525 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d0649bcab0441497de832d0b77d44b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35575 } }
I20260924 23:20:05.452523 22525 leader_election.cc:302] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2d0649bcab0441497de832d0b77d44b; no voters:
I20260924 23:20:05.452690 22525 leader_election.cc:288] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:20:05.452759 22530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:20:05.452909 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:20:05.452943 22525 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:05.454177 22530 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [term 1 LEADER]: Becoming Leader. State: Replica: c2d0649bcab0441497de832d0b77d44b, State: Running, Role: LEADER
I20260924 23:20:05.454365 22530 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d0649bcab0441497de832d0b77d44b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35575 } }
I20260924 23:20:05.454869 22546 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2d0649bcab0441497de832d0b77d44b. Latest consensus state: current_term: 1 leader_uuid: "c2d0649bcab0441497de832d0b77d44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d0649bcab0441497de832d0b77d44b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35575 } } }
I20260924 23:20:05.454916 22531 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2d0649bcab0441497de832d0b77d44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d0649bcab0441497de832d0b77d44b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35575 } } }
I20260924 23:20:05.455582 22531 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [sys.catalog]: This master's current role is: LEADER
W20260924 23:20:05.455597 22549 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:20:05.455665 22549 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:20:05.455011 22546 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:05.455924 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:05.454321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:55157 (local address 127.0.0.1:35575)
0924 23:20:05.454353 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:05.454355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:05.454772 (+ 417us) server_negotiation.cc:407] Connection header received
0924 23:20:05.454873 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:05.454874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:05.454880 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:05.454907 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:05.455653 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:05.455654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:05.455655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:20:05.455659 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:05.455677 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:20:05.455682 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:05.455736 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:05.455858 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 23:20:05.455870 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 23:20:05.456053 22552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:20:05.456375 22552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:20:05.457912 22552 catalog_manager.cc:1347] Generated new cluster ID: b4fbbd24b7b84f598036295d3e2a481d
I20260924 23:20:05.457963 22552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:20:05.467866 22552 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:20:05.468055 22552 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:20:05.472779 22552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2d0649bcab0441497de832d0b77d44b: Generated new TSK 0
I20260924 23:20:05.472980 22552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:20:05.480049 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 22457
2026-09-24T23:20:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:20:05 dist-test-slave-s1hw krb5kdc[22556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:20:05 dist-test-slave-s1hw krb5kdc[22556](info): set up 2 sockets
Sep 24 23:20:05 dist-test-slave-s1hw krb5kdc[22556](info): commencing operation
krb5kdc: starting...
W20260924 23:20:06.431324 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:20:06 dist-test-slave-s1hw krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:20:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:20:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:20:06.550824 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:45559
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:35611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:45559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:20:06.630342 22569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:06.630579 22569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:06.630602 22569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:20:06.631876 22569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:06.631922 22569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:06.631937 22569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:06.631947 22569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:06.631959 22569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:20:06.633510 22569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:35611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:45559
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:45559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:06.633757 22569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:06.633962 22569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:06.638808 22574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:06.638881 22577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:06.638922 22569 server_base.cc:1034] running on GCE node
W20260924 23:20:06.638798 22575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:06.639272 22569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:06.639557 22569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:06.640698 22569 hybrid_clock.cc:648] HybridClock initialized: now 1790292006640678 us; error 34 us; skew 500 ppm
Sep 24 23:20:06 dist-test-slave-s1hw krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292006, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:06.643076 22569 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:20:06.643596 22569 webserver.cc:460] Webserver started at http://127.1.4.254:39785/ using document root <none> and password file <none>
I20260924 23:20:06.643807 22569 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:06.643854 22569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:06.643981 22569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:06.645056 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "5dc45cec1064453caee53cac75303be3"
format_stamp: "Formatted at 2026-09-24 23:20:06 on dist-test-slave-s1hw"
server_key: "ddef33bd941ebe54d57dc2fa509ffe60"
server_key_iv: "4466013db47a4a4f894b5e656643d213"
server_key_version: "encryptionkey@0"
I20260924 23:20:06.645459 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "5dc45cec1064453caee53cac75303be3"
format_stamp: "Formatted at 2026-09-24 23:20:06 on dist-test-slave-s1hw"
server_key: "ddef33bd941ebe54d57dc2fa509ffe60"
server_key_iv: "4466013db47a4a4f894b5e656643d213"
server_key_version: "encryptionkey@0"
I20260924 23:20:06.646852 22569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:20:06.647691 22584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:06.647848 22569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:20:06.647930 22569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "5dc45cec1064453caee53cac75303be3"
format_stamp: "Formatted at 2026-09-24 23:20:06 on dist-test-slave-s1hw"
server_key: "ddef33bd941ebe54d57dc2fa509ffe60"
server_key_iv: "4466013db47a4a4f894b5e656643d213"
server_key_version: "encryptionkey@0"
I20260924 23:20:06.647996 22569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:06.660907 22569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:06.663069 22569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:06.663196 22569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:06.666911 22569 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:45559
I20260924 23:20:06.666978 22636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:45559 every 8 connection(s)
I20260924 23:20:06.667289 22569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:20:06.668202 22637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:06.671095 22637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3: Bootstrap starting.
I20260924 23:20:06.671758 22637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:06.672065 22637 log.cc:824] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:06.672729 22637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3: No bootstrap required, opened a new log
I20260924 23:20:06.674332 22637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc45cec1064453caee53cac75303be3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45559 } }
I20260924 23:20:06.674476 22637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:20:06.674497 22637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc45cec1064453caee53cac75303be3, State: Initialized, Role: FOLLOWER
I20260924 23:20:06.674642 22637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc45cec1064453caee53cac75303be3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45559 } }
I20260924 23:20:06.674708 22637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:20:06.674729 22637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:20:06.674755 22637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:20:06.675601 22637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc45cec1064453caee53cac75303be3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45559 } }
I20260924 23:20:06.675701 22637 leader_election.cc:302] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dc45cec1064453caee53cac75303be3; no voters:
I20260924 23:20:06.675732 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 22569
I20260924 23:20:06.675853 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:20:06.675859 22637 leader_election.cc:288] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:20:06.675882 22640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:20:06.675987 22640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [term 1 LEADER]: Becoming Leader. State: Replica: 5dc45cec1064453caee53cac75303be3, State: Running, Role: LEADER
I20260924 23:20:06.676067 1043 external_mini_cluster.cc:1428] Setting key f7c51997be34947eff57e8d07ab5d44a
I20260924 23:20:06.676103 22637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:06.676118 22640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc45cec1064453caee53cac75303be3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45559 } }
I20260924 23:20:06.676545 22642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dc45cec1064453caee53cac75303be3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc45cec1064453caee53cac75303be3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45559 } } }
I20260924 23:20:06.676656 22642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:06.676677 22641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dc45cec1064453caee53cac75303be3. Latest consensus state: current_term: 1 leader_uuid: "5dc45cec1064453caee53cac75303be3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dc45cec1064453caee53cac75303be3" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 45559 } } }
I20260924 23:20:06.676782 22641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:06.677654 22651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:20:06.678851 22651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:20:06.680554 22651 catalog_manager.cc:1347] Generated new cluster ID: a716ca51c20c4030b30832652564c176
I20260924 23:20:06.680611 22651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:20:06 dist-test-slave-s1hw krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:20:06.688198 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:06.676609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52971 (local address 127.1.4.254:45559)
0924 23:20:06.676864 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:06.676874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:06.677019 (+ 145us) server_negotiation.cc:407] Connection header received
0924 23:20:06.677148 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:06.677156 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:06.677255 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:06.678246 (+ 991us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:06.678362 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:06.679716 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:06.680729 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:06.681367 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:06.681486 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:06.684414 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:06.684419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:06.684419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:06.684433 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:06.686401 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:06.686851 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:06.686854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:06.686855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:06.686975 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:06.687214 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:06.687214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:06.687215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:06.687565 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:06.687733 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:06.687863 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 23:20:06.687964 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":91,"threads_started":1}
I20260924 23:20:06.689311 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:20:06.690965 22651 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:20:06.691707 22651 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:20:06.698992 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:06.690803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53375 (local address 127.1.4.254:45559)
0924 23:20:06.690835 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:06.690837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:06.690960 (+ 123us) server_negotiation.cc:407] Connection header received
0924 23:20:06.691052 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:06.691053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:06.691063 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:06.691110 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:06.691727 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:06.692229 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:06.692990 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:06.693222 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:06.693279 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:06.696406 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:06.696408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:06.696408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:06.696415 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:06.697665 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:06.697992 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:06.697993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:06.697994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:06.698039 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:06.698232 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:06.698233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:06.698234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:06.698332 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:06.698827 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:06.698868 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 23:20:06.698914 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 23:20:06.699503 22651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dc45cec1064453caee53cac75303be3: Generated new TSK 0
I20260924 23:20:06.699651 22651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:20:06.703600 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 22569
W20260924 23:20:06.707451 22554 connection.cc:535] client connection to 127.1.4.254:45559 recv error: Network error: failed to read from TLS socket (remote: 127.1.4.254:45559): Connection reset by peer (error 104)
2026-09-24T23:20:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:20:06 dist-test-slave-s1hw krb5kdc[22668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:20:06 dist-test-slave-s1hw krb5kdc[22668](info): set up 2 sockets
Sep 24 23:20:06 dist-test-slave-s1hw krb5kdc[22668](info): commencing operation
krb5kdc: starting...
W20260924 23:20:07.651182 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:20:07 dist-test-slave-s1hw krb5kdc[22668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:20:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:20:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:20:07.769876 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:33979
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:33979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:20:07.849936 22681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:07.850167 22681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:07.850193 22681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:20:07.851388 22681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:07.851446 22681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:07.851461 22681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:07.851472 22681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:07.851483 22681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:20:07.853035 22681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:33979
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:33979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:07.853267 22681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:07.853452 22681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:07.858319 22686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:07.858330 22689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:07.858373 22681 server_base.cc:1034] running on GCE node
W20260924 23:20:07.858330 22687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:07.858840 22681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:07.859136 22681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:07.860292 22681 hybrid_clock.cc:648] HybridClock initialized: now 1790292007860278 us; error 31 us; skew 500 ppm
Sep 24 23:20:07 dist-test-slave-s1hw krb5kdc[22668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292007, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:07.862759 22681 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:20:07.863195 22681 webserver.cc:460] Webserver started at http://127.1.4.254:39399/ using document root <none> and password file <none>
I20260924 23:20:07.863440 22681 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:07.863494 22681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:07.863647 22681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:07.864697 22681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "4310a9d8f68f4b6cbefacf7d40de7299"
format_stamp: "Formatted at 2026-09-24 23:20:07 on dist-test-slave-s1hw"
server_key: "1577aa17db7c938b1d718ae9504bbacc"
server_key_iv: "d12faaa173b4014d0c88827b06d5e04a"
server_key_version: "encryptionkey@0"
I20260924 23:20:07.865031 22681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "4310a9d8f68f4b6cbefacf7d40de7299"
format_stamp: "Formatted at 2026-09-24 23:20:07 on dist-test-slave-s1hw"
server_key: "1577aa17db7c938b1d718ae9504bbacc"
server_key_iv: "d12faaa173b4014d0c88827b06d5e04a"
server_key_version: "encryptionkey@0"
I20260924 23:20:07.866353 22681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:20:07.867096 22696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:07.867266 22681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:20:07.867352 22681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "4310a9d8f68f4b6cbefacf7d40de7299"
format_stamp: "Formatted at 2026-09-24 23:20:07 on dist-test-slave-s1hw"
server_key: "1577aa17db7c938b1d718ae9504bbacc"
server_key_iv: "d12faaa173b4014d0c88827b06d5e04a"
server_key_version: "encryptionkey@0"
I20260924 23:20:07.867460 22681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:07.886778 22681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:07.889050 22681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:07.889189 22681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:07.893023 22681 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:33979
I20260924 23:20:07.893059 22748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:33979 every 8 connection(s)
I20260924 23:20:07.893390 22681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:20:07.894254 22749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:07.894589 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 22681
I20260924 23:20:07.894717 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:20:07.894902 1043 external_mini_cluster.cc:1428] Setting key 3f5d803df156b9a1375ba0c37a6190e6
I20260924 23:20:07.897287 22749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299: Bootstrap starting.
I20260924 23:20:07.897969 22749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:07.898257 22749 log.cc:824] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:07.898897 22749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299: No bootstrap required, opened a new log
I20260924 23:20:07.900941 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:07.895439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48185 (local address 127.1.4.254:33979)
0924 23:20:07.895629 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:07.895639 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:07.895675 (+ 36us) server_negotiation.cc:407] Connection header received
0924 23:20:07.896205 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:07.896214 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:07.896277 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:07.896435 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:07.896629 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:07.897972 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:07.898843 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:07.899448 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:07.899567 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:07.900206 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:07.900211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:07.900211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:20:07.900413 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:07.900452 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:20:07.900467 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:07.900556 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:07.900673 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 23:20:07.900711 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260924 23:20:07.901319 22749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4310a9d8f68f4b6cbefacf7d40de7299" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33979 } }
I20260924 23:20:07.901489 22749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:20:07.901523 22749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4310a9d8f68f4b6cbefacf7d40de7299, State: Initialized, Role: FOLLOWER
I20260924 23:20:07.901639 22749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4310a9d8f68f4b6cbefacf7d40de7299" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33979 } }
I20260924 23:20:07.901715 22749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:20:07.901747 22749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:20:07.901793 22749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:20:07.902022 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:20:07.902410 22749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4310a9d8f68f4b6cbefacf7d40de7299" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33979 } }
I20260924 23:20:07.902505 22749 leader_election.cc:302] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4310a9d8f68f4b6cbefacf7d40de7299; no voters:
I20260924 23:20:07.902663 22749 leader_election.cc:288] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:20:07.902727 22754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:20:07.902908 22749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:07.902930 22754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [term 1 LEADER]: Becoming Leader. State: Replica: 4310a9d8f68f4b6cbefacf7d40de7299, State: Running, Role: LEADER
I20260924 23:20:07.903174 22754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4310a9d8f68f4b6cbefacf7d40de7299" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33979 } }
I20260924 23:20:07.903810 22755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4310a9d8f68f4b6cbefacf7d40de7299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4310a9d8f68f4b6cbefacf7d40de7299" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33979 } } }
I20260924 23:20:07.903900 22755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:07.904059 22757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4310a9d8f68f4b6cbefacf7d40de7299. Latest consensus state: current_term: 1 leader_uuid: "4310a9d8f68f4b6cbefacf7d40de7299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4310a9d8f68f4b6cbefacf7d40de7299" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33979 } } }
I20260924 23:20:07.904115 22757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:07.904100 22768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:20:07.904623 22768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:20:07.906327 22768 catalog_manager.cc:1347] Generated new cluster ID: 95fa4b4570eb499086d3dd538bd873a3
I20260924 23:20:07.906379 22768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:20:07.907208 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:07.903806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35129 (local address 127.1.4.254:33979)
0924 23:20:07.903840 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:07.903842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:07.903850 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:20:07.903869 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:07.903870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:07.903880 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:07.903934 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:07.904126 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:07.904811 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:07.905762 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:07.906042 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:07.906169 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:07.906982 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:07.906983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:07.906984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:20:07.906988 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:07.907003 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:20:07.907008 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:07.907053 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:07.907130 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 23:20:07.907140 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:20:07.918098 22768 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:20:07.918841 22768 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:20:07.931545 22768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4310a9d8f68f4b6cbefacf7d40de7299: Generated new TSK 0
I20260924 23:20:07.931804 22768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:20:07.936167 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 22681
W20260924 23:20:07.939940 22666 connection.cc:535] client connection to 127.1.4.254:33979 recv error: Network error: failed to read from TLS socket (remote: 127.1.4.254:33979): Connection reset by peer (error 104)
2026-09-24T23:20:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1232 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:20:07 dist-test-slave-s1hw krb5kdc[22780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:20:07 dist-test-slave-s1hw krb5kdc[22780](info): set up 2 sockets
Sep 24 23:20:07 dist-test-slave-s1hw krb5kdc[22780](info): commencing operation
krb5kdc: starting...
W20260924 23:20:08.887246 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:20:08 dist-test-slave-s1hw krb5kdc[22780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:20:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:20:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:20:09.006387 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:41965
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:41965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:20:09.087390 22793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:09.087608 22793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:09.087639 22793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:20:09.088816 22793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:09.088860 22793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:09.088878 22793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:09.088894 22793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:09.088908 22793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:20:09.090391 22793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:41965
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:09.090668 22793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:09.090876 22793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:09.095707 22799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:09.095701 22801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:09.095722 22798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:09.095765 22793 server_base.cc:1034] running on GCE node
I20260924 23:20:09.096101 22793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:09.096366 22793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:09.097527 22793 hybrid_clock.cc:648] HybridClock initialized: now 1790292009097512 us; error 28 us; skew 500 ppm
Sep 24 23:20:09 dist-test-slave-s1hw krb5kdc[22780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292009, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:09.099911 22793 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:20:09.100329 22793 webserver.cc:460] Webserver started at http://127.1.4.254:40905/ using document root <none> and password file <none>
I20260924 23:20:09.100550 22793 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:09.100610 22793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:09.100733 22793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:09.101749 22793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "f7b3bdc1ca8840f4bd822a32a72d9318"
format_stamp: "Formatted at 2026-09-24 23:20:09 on dist-test-slave-s1hw"
server_key: "52bfad1c25da787f11b8734862e5623f"
server_key_iv: "e567d3a2e9b25da373032ba5597e01f6"
server_key_version: "encryptionkey@0"
I20260924 23:20:09.102061 22793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "f7b3bdc1ca8840f4bd822a32a72d9318"
format_stamp: "Formatted at 2026-09-24 23:20:09 on dist-test-slave-s1hw"
server_key: "52bfad1c25da787f11b8734862e5623f"
server_key_iv: "e567d3a2e9b25da373032ba5597e01f6"
server_key_version: "encryptionkey@0"
I20260924 23:20:09.103281 22793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:20:09.104072 22808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:09.104205 22793 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:20:09.104282 22793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "f7b3bdc1ca8840f4bd822a32a72d9318"
format_stamp: "Formatted at 2026-09-24 23:20:09 on dist-test-slave-s1hw"
server_key: "52bfad1c25da787f11b8734862e5623f"
server_key_iv: "e567d3a2e9b25da373032ba5597e01f6"
server_key_version: "encryptionkey@0"
I20260924 23:20:09.104347 22793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:09.107546 22793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:09.109643 22793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:09.109754 22793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:09.113435 22793 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:41965
I20260924 23:20:09.113471 22860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:41965 every 8 connection(s)
I20260924 23:20:09.113780 22793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:20:09.114701 22861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:09.117003 22861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318: Bootstrap starting.
I20260924 23:20:09.117553 22861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:09.117806 22861 log.cc:824] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:09.118484 22861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318: No bootstrap required, opened a new log
I20260924 23:20:09.120074 22861 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b3bdc1ca8840f4bd822a32a72d9318" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41965 } }
I20260924 23:20:09.120252 22861 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:20:09.120276 22861 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b3bdc1ca8840f4bd822a32a72d9318, State: Initialized, Role: FOLLOWER
I20260924 23:20:09.120407 22861 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b3bdc1ca8840f4bd822a32a72d9318" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41965 } }
I20260924 23:20:09.120469 22861 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:20:09.120489 22861 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:20:09.120533 22861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:20:09.121001 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 22793
I20260924 23:20:09.121122 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:20:09.121318 1043 external_mini_cluster.cc:1428] Setting key 789587360ff052553b92596248cf4815
I20260924 23:20:09.121531 22861 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b3bdc1ca8840f4bd822a32a72d9318" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41965 } }
I20260924 23:20:09.121622 22861 leader_election.cc:302] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7b3bdc1ca8840f4bd822a32a72d9318; no voters:
I20260924 23:20:09.121809 22861 leader_election.cc:288] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:20:09.121860 22864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:20:09.122049 22864 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [term 1 LEADER]: Becoming Leader. State: Replica: f7b3bdc1ca8840f4bd822a32a72d9318, State: Running, Role: LEADER
I20260924 23:20:09.122068 22861 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:09.122231 22864 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b3bdc1ca8840f4bd822a32a72d9318" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41965 } }
I20260924 23:20:09.123286 22867 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7b3bdc1ca8840f4bd822a32a72d9318. Latest consensus state: current_term: 1 leader_uuid: "f7b3bdc1ca8840f4bd822a32a72d9318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b3bdc1ca8840f4bd822a32a72d9318" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41965 } } }
I20260924 23:20:09.123478 22867 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:09.123960 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:09.121768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46405 (local address 127.1.4.254:41965)
0924 23:20:09.121940 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:09.121950 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:09.122059 (+ 109us) server_negotiation.cc:407] Connection header received
0924 23:20:09.122247 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:09.122256 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:09.122289 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:09.122397 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:09.123416 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:09.123421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:09.123421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:20:09.123427 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:09.123475 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:20:09.123493 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:09.123577 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:09.123776 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 23:20:09.123819 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
W20260924 23:20:09.124161 22882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 23:20:09.124184 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7b3bdc1ca8840f4bd822a32a72d9318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b3bdc1ca8840f4bd822a32a72d9318" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41965 } } }
I20260924 23:20:09.124523 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318 [sys.catalog]: This master's current role is: LEADER
W20260924 23:20:09.124471 22882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:20:09.124656 22881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:20:09.124809 22881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:20:09.125066 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:20:09.127502 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:09.126511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48949 (local address 127.1.4.254:41965)
0924 23:20:09.126549 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:09.126551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:09.126685 (+ 134us) server_negotiation.cc:407] Connection header received
0924 23:20:09.126736 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:09.126737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:09.126742 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:09.126780 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:09.127280 (+ 500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:09.127282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:09.127282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:20:09.127286 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:09.127300 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:20:09.127304 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:09.127348 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:09.127428 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 23:20:09.127438 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:20:09.131841 22881 catalog_manager.cc:1347] Generated new cluster ID: c8a6437aac444ea88718281e8dfd9dda
I20260924 23:20:09.131904 22881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:20:09.164875 22881 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:20:09.165110 22881 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:20:09.170377 22881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7b3bdc1ca8840f4bd822a32a72d9318: Generated new TSK 0
I20260924 23:20:09.170574 22881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:20:09.179862 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 22793
2026-09-24T23:20:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1243 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:20:09 dist-test-slave-s1hw krb5kdc[22892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:20:09 dist-test-slave-s1hw krb5kdc[22892](info): set up 2 sockets
Sep 24 23:20:09 dist-test-slave-s1hw krb5kdc[22892](info): commencing operation
krb5kdc: starting...
W20260924 23:20:10.143277 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:20:10 dist-test-slave-s1hw krb5kdc[22892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:20:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:20:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:20:10.262642 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:33837
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:33837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:20:10.343894 22905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:10.344120 22905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:10.344142 22905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:20:10.345347 22905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:10.345394 22905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:10.345407 22905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:10.345418 22905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:10.345428 22905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:20:10.346982 22905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:33837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:33837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:10.347214 22905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:10.347438 22905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:10.352356 22910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:10.352398 22905 server_base.cc:1034] running on GCE node
W20260924 23:20:10.352356 22911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:10.352330 22913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:10.352887 22905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:10.353189 22905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:10.354357 22905 hybrid_clock.cc:648] HybridClock initialized: now 1790292010354339 us; error 28 us; skew 500 ppm
Sep 24 23:20:10 dist-test-slave-s1hw krb5kdc[22892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:10.356848 22905 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:20:10.357338 22905 webserver.cc:460] Webserver started at http://127.1.4.254:37411/ using document root <none> and password file <none>
I20260924 23:20:10.357563 22905 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:10.357612 22905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:10.357735 22905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:10.358829 22905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "4e3e2707c39841da8b82cdf430e76440"
format_stamp: "Formatted at 2026-09-24 23:20:10 on dist-test-slave-s1hw"
server_key: "fcb0ef13ac17c23fbf2fd8afd7129c45"
server_key_iv: "d23fe3a0b9d2f5d6a3ca75ade4dc336b"
server_key_version: "encryptionkey@0"
I20260924 23:20:10.359156 22905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "4e3e2707c39841da8b82cdf430e76440"
format_stamp: "Formatted at 2026-09-24 23:20:10 on dist-test-slave-s1hw"
server_key: "fcb0ef13ac17c23fbf2fd8afd7129c45"
server_key_iv: "d23fe3a0b9d2f5d6a3ca75ade4dc336b"
server_key_version: "encryptionkey@0"
I20260924 23:20:10.360486 22905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:20:10.361243 22920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:10.361418 22905 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:20:10.361503 22905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "4e3e2707c39841da8b82cdf430e76440"
format_stamp: "Formatted at 2026-09-24 23:20:10 on dist-test-slave-s1hw"
server_key: "fcb0ef13ac17c23fbf2fd8afd7129c45"
server_key_iv: "d23fe3a0b9d2f5d6a3ca75ade4dc336b"
server_key_version: "encryptionkey@0"
I20260924 23:20:10.361572 22905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:10.377934 22905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:10.380241 22905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:10.380393 22905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:10.384344 22905 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:33837
I20260924 23:20:10.384436 22972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:33837 every 8 connection(s)
I20260924 23:20:10.384733 22905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:20:10.385618 22973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:10.387475 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 22905
I20260924 23:20:10.387605 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:20:10.387789 1043 external_mini_cluster.cc:1428] Setting key d69ac539863de8159505f285fd38b66f
I20260924 23:20:10.388235 22973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440: Bootstrap starting.
I20260924 23:20:10.388880 22973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:10.389168 22973 log.cc:824] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:10.389895 22973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440: No bootstrap required, opened a new log
I20260924 23:20:10.391793 22973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3e2707c39841da8b82cdf430e76440" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33837 } }
I20260924 23:20:10.391935 22973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:20:10.391968 22973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3e2707c39841da8b82cdf430e76440, State: Initialized, Role: FOLLOWER
I20260924 23:20:10.392091 22973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3e2707c39841da8b82cdf430e76440" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33837 } }
I20260924 23:20:10.392153 22973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:20:10.392194 22973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:20:10.392283 22973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:20:10.392949 22973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3e2707c39841da8b82cdf430e76440" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33837 } }
I20260924 23:20:10.393064 22973 leader_election.cc:302] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e3e2707c39841da8b82cdf430e76440; no voters:
I20260924 23:20:10.393239 22973 leader_election.cc:288] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:20:10.393330 22978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:20:10.393544 22973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:10.393563 22978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [term 1 LEADER]: Becoming Leader. State: Replica: 4e3e2707c39841da8b82cdf430e76440, State: Running, Role: LEADER
I20260924 23:20:10.393734 22978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3e2707c39841da8b82cdf430e76440" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33837 } }
I20260924 23:20:10.394294 22978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3e2707c39841da8b82cdf430e76440. Latest consensus state: current_term: 1 leader_uuid: "4e3e2707c39841da8b82cdf430e76440" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3e2707c39841da8b82cdf430e76440" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33837 } } }
I20260924 23:20:10.394428 22978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [sys.catalog]: This master's current role is: LEADER
Sep 24 23:20:10 dist-test-slave-s1hw krb5kdc[22892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:20:10.394291 22979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e3e2707c39841da8b82cdf430e76440" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3e2707c39841da8b82cdf430e76440" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 33837 } } }
I20260924 23:20:10.395051 22979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:10.394951 22989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:20:10.395797 22989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:20:10.397626 22989 catalog_manager.cc:1347] Generated new cluster ID: acc419fcecc6459ea1acfc51389e5d50
I20260924 23:20:10.397682 22989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:20:10.398860 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:10.388342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60119 (local address 127.1.4.254:33837)
0924 23:20:10.388525 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:10.388534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:10.388560 (+ 26us) server_negotiation.cc:407] Connection header received
0924 23:20:10.389034 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:10.389042 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:10.389095 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:10.389239 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:10.389928 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:10.390862 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:10.391684 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:10.392158 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:10.392280 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:10.395098 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:10.395103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:10.395104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:10.395117 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:10.397186 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:10.397635 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:10.397638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:10.397639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:10.397745 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:10.397943 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:10.397944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:10.397945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:10.398250 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:10.398384 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:10.398526 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 23:20:10.398656 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260924 23:20:10.399840 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:20:10.407799 22989 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:20:10.408516 22989 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:20:10.415081 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:10.407564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54173 (local address 127.1.4.254:33837)
0924 23:20:10.407629 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:10.407631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:10.407928 (+ 297us) server_negotiation.cc:407] Connection header received
0924 23:20:10.408018 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:10.408019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:10.408030 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:10.408104 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:10.408761 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:10.409528 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:10.410536 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:10.410873 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:10.410947 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:10.412394 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:10.412396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:10.412396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:10.412404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:10.413966 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:10.414334 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:10.414335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:10.414335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:10.414397 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:10.414565 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:10.414565 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:10.414565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:10.414704 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:10.414796 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:10.414884 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 23:20:10.414979 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 23:20:10.417093 22989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e3e2707c39841da8b82cdf430e76440: Generated new TSK 0
I20260924 23:20:10.417292 22989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:20:10.422148 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 22905
2026-09-24T23:20:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1242 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:20:10 dist-test-slave-s1hw krb5kdc[23004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:20:10 dist-test-slave-s1hw krb5kdc[23004](info): set up 2 sockets
Sep 24 23:20:10 dist-test-slave-s1hw krb5kdc[23004](info): commencing operation
krb5kdc: starting...
W20260924 23:20:11.387339 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:20:11 dist-test-slave-s1hw krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:20:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:20:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:20:11.504326 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38389
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:38389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:20:11.584740 23017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:11.584962 23017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:11.584985 23017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:20:11.586141 23017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:11.586181 23017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:11.586194 23017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:11.586205 23017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:11.586216 23017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:20:11.587695 23017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:38389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:11.587899 23017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:11.588086 23017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:11.592972 23023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:11.592993 23025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:11.593015 23022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:11.593021 23017 server_base.cc:1034] running on GCE node
I20260924 23:20:11.593448 23017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:11.593741 23017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:11.594913 23017 hybrid_clock.cc:648] HybridClock initialized: now 1790292011594893 us; error 28 us; skew 500 ppm
Sep 24 23:20:11 dist-test-slave-s1hw krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292011, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:11.597682 23017 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:20:11.598114 23017 webserver.cc:460] Webserver started at http://127.1.4.254:42265/ using document root <none> and password file <none>
I20260924 23:20:11.598326 23017 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:11.598373 23017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:11.598497 23017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:11.599612 23017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "a63dbf5925dd46bdb004d0894bc05fe7"
format_stamp: "Formatted at 2026-09-24 23:20:11 on dist-test-slave-s1hw"
server_key: "0c423eff36da246ba0f1cc3ad9b29751"
server_key_iv: "6b17bdc8c625b7b822bb9c4fa00fdbf4"
server_key_version: "encryptionkey@0"
I20260924 23:20:11.599929 23017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "a63dbf5925dd46bdb004d0894bc05fe7"
format_stamp: "Formatted at 2026-09-24 23:20:11 on dist-test-slave-s1hw"
server_key: "0c423eff36da246ba0f1cc3ad9b29751"
server_key_iv: "6b17bdc8c625b7b822bb9c4fa00fdbf4"
server_key_version: "encryptionkey@0"
I20260924 23:20:11.601225 23017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:20:11.602051 23032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:11.602259 23017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:20:11.602349 23017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "a63dbf5925dd46bdb004d0894bc05fe7"
format_stamp: "Formatted at 2026-09-24 23:20:11 on dist-test-slave-s1hw"
server_key: "0c423eff36da246ba0f1cc3ad9b29751"
server_key_iv: "6b17bdc8c625b7b822bb9c4fa00fdbf4"
server_key_version: "encryptionkey@0"
I20260924 23:20:11.602409 23017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:11.638087 23017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:11.640836 23017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:11.641003 23017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:11.645689 23017 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:38389
I20260924 23:20:11.645735 23084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:38389 every 8 connection(s)
I20260924 23:20:11.646078 23017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:20:11.646999 23085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:11.649281 23085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7: Bootstrap starting.
I20260924 23:20:11.649518 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 23017
I20260924 23:20:11.649618 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:20:11.649782 1043 external_mini_cluster.cc:1428] Setting key 266814d51cf00e418adbe610f398bd7b
I20260924 23:20:11.649866 23085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:11.650182 23085 log.cc:824] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:11.651067 23085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7: No bootstrap required, opened a new log
I20260924 23:20:11.653116 23085 raft_consensus.cc:348] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63dbf5925dd46bdb004d0894bc05fe7" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38389 } }
I20260924 23:20:11.653288 23085 raft_consensus.cc:374] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:20:11.653319 23085 raft_consensus.cc:729] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a63dbf5925dd46bdb004d0894bc05fe7, State: Initialized, Role: FOLLOWER
I20260924 23:20:11.653439 23085 consensus_queue.cc:260] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63dbf5925dd46bdb004d0894bc05fe7" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38389 } }
I20260924 23:20:11.653505 23085 raft_consensus.cc:388] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:20:11.653540 23085 raft_consensus.cc:482] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:20:11.653586 23085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:20:11.654119 23085 raft_consensus.cc:504] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63dbf5925dd46bdb004d0894bc05fe7" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38389 } }
I20260924 23:20:11.654229 23085 leader_election.cc:302] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a63dbf5925dd46bdb004d0894bc05fe7; no voters:
I20260924 23:20:11.654399 23085 leader_election.cc:288] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:20:11.654423 23090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:20:11.654596 23090 raft_consensus.cc:686] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [term 1 LEADER]: Becoming Leader. State: Replica: a63dbf5925dd46bdb004d0894bc05fe7, State: Running, Role: LEADER
I20260924 23:20:11.654840 23090 consensus_queue.cc:237] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63dbf5925dd46bdb004d0894bc05fe7" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38389 } }
I20260924 23:20:11.654712 23085 sys_catalog.cc:564] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:11.655264 23092 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a63dbf5925dd46bdb004d0894bc05fe7. Latest consensus state: current_term: 1 leader_uuid: "a63dbf5925dd46bdb004d0894bc05fe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63dbf5925dd46bdb004d0894bc05fe7" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38389 } } }
I20260924 23:20:11.655491 23092 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:11.655426 23091 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a63dbf5925dd46bdb004d0894bc05fe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63dbf5925dd46bdb004d0894bc05fe7" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38389 } } }
I20260924 23:20:11.655720 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:11.650345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38969 (local address 127.1.4.254:38389)
0924 23:20:11.650555 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:11.650566 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:11.650600 (+ 34us) server_negotiation.cc:407] Connection header received
0924 23:20:11.651082 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:11.651090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:11.651155 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:11.651285 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:11.651608 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:11.652537 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:11.653617 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:11.654068 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:11.654151 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:11.654874 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:11.654879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:11.654879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:20:11.655079 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:11.655123 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:20:11.655140 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:11.655209 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:11.655457 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 23:20:11.655524 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":98,"threads_started":1}
I20260924 23:20:11.655759 23091 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:11.656606 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 23:20:11.657186 23106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:20:11.657275 23106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:20:11.657392 23094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:20:11.657661 23094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:20:11.659224 23094 catalog_manager.cc:1347] Generated new cluster ID: 0c5e1f11b70548e3a69778cc2e240e3f
I20260924 23:20:11.659269 23094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:20:11.660913 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:11.657989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43201 (local address 127.1.4.254:38389)
0924 23:20:11.658028 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:11.658029 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:11.658071 (+ 42us) server_negotiation.cc:407] Connection header received
0924 23:20:11.658143 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:11.658144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:11.658154 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:11.658206 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:11.658326 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:11.659010 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:11.659847 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:11.660054 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:11.660108 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:11.660694 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:11.660696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:11.660696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:20:11.660700 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:11.660715 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:20:11.660719 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:11.660763 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:11.660826 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 23:20:11.660845 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:20:11.692240 23094 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:20:11.692873 23094 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:20:11.700706 23094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a63dbf5925dd46bdb004d0894bc05fe7: Generated new TSK 0
I20260924 23:20:11.700958 23094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:20:11.711479 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 23017
2026-09-24T23:20:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:20:11 dist-test-slave-s1hw krb5kdc[23116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:20:11 dist-test-slave-s1hw krb5kdc[23116](info): set up 2 sockets
Sep 24 23:20:11 dist-test-slave-s1hw krb5kdc[23116](info): commencing operation
krb5kdc: starting...
W20260924 23:20:12.663208 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:20:12 dist-test-slave-s1hw krb5kdc[23116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:20:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:20:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:20:12.783955 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38809
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:38809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:20:12.866431 23129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:12.866662 23129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:12.866685 23129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:20:12.867940 23129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:12.867985 23129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:12.868000 23129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:12.868010 23129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:12.868021 23129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:20:12.869594 23129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:38809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:38809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:12.869798 23129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:12.869976 23129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:12.874702 23134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:12.874900 23129 server_base.cc:1034] running on GCE node
W20260924 23:20:12.874708 23137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:12.874706 23135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:12.875253 23129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:12.875563 23129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:12.876714 23129 hybrid_clock.cc:648] HybridClock initialized: now 1790292012876695 us; error 35 us; skew 500 ppm
Sep 24 23:20:12 dist-test-slave-s1hw krb5kdc[23116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292012, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:12.879319 23129 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:20:12.879773 23129 webserver.cc:460] Webserver started at http://127.1.4.254:41561/ using document root <none> and password file <none>
I20260924 23:20:12.879956 23129 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:12.879994 23129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:12.880081 23129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:12.881115 23129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "d5cce45d46304abaac60dcd47b4e298a"
format_stamp: "Formatted at 2026-09-24 23:20:12 on dist-test-slave-s1hw"
server_key: "7e3d1717477685f4ef7291e5c779e1b0"
server_key_iv: "b2b812c3448d240c172a5092d8d11b6e"
server_key_version: "encryptionkey@0"
I20260924 23:20:12.881434 23129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "d5cce45d46304abaac60dcd47b4e298a"
format_stamp: "Formatted at 2026-09-24 23:20:12 on dist-test-slave-s1hw"
server_key: "7e3d1717477685f4ef7291e5c779e1b0"
server_key_iv: "b2b812c3448d240c172a5092d8d11b6e"
server_key_version: "encryptionkey@0"
I20260924 23:20:12.882673 23129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 23:20:12.883339 23144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:12.883528 23129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:12.883605 23129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "d5cce45d46304abaac60dcd47b4e298a"
format_stamp: "Formatted at 2026-09-24 23:20:12 on dist-test-slave-s1hw"
server_key: "7e3d1717477685f4ef7291e5c779e1b0"
server_key_iv: "b2b812c3448d240c172a5092d8d11b6e"
server_key_version: "encryptionkey@0"
I20260924 23:20:12.883675 23129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:12.886907 23129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:12.889091 23129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:12.889225 23129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:12.892807 23129 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:38809
I20260924 23:20:12.892838 23196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:38809 every 8 connection(s)
I20260924 23:20:12.893149 23129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:20:12.894109 23197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:12.896600 23197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a: Bootstrap starting.
I20260924 23:20:12.897130 23197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:12.897352 23197 log.cc:824] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:12.898120 23197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a: No bootstrap required, opened a new log
I20260924 23:20:12.898669 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 23129
I20260924 23:20:12.898788 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:20:12.898969 1043 external_mini_cluster.cc:1428] Setting key 54173d3d6d5cafdec558bbcfed53cb9a
I20260924 23:20:12.900270 23197 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cce45d46304abaac60dcd47b4e298a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38809 } }
I20260924 23:20:12.900418 23197 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:20:12.900452 23197 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5cce45d46304abaac60dcd47b4e298a, State: Initialized, Role: FOLLOWER
I20260924 23:20:12.900630 23197 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cce45d46304abaac60dcd47b4e298a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38809 } }
I20260924 23:20:12.900720 23197 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:20:12.900745 23197 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:20:12.900791 23197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:20:12.901480 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:12.899507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56109 (local address 127.1.4.254:38809)
0924 23:20:12.899684 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:12.899692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:12.899717 (+ 25us) server_negotiation.cc:407] Connection header received
0924 23:20:12.900247 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:12.900256 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:12.900291 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:12.900421 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:12.901073 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:12.901076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:12.901076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:20:12.901079 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:12.901110 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:20:12.901120 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:12.901177 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:12.901248 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 23:20:12.901281 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260924 23:20:12.901829 23197 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cce45d46304abaac60dcd47b4e298a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38809 } }
I20260924 23:20:12.901928 23197 leader_election.cc:302] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5cce45d46304abaac60dcd47b4e298a; no voters:
I20260924 23:20:12.902094 23197 leader_election.cc:288] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:20:12.902125 23202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:20:12.902277 23202 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [term 1 LEADER]: Becoming Leader. State: Replica: d5cce45d46304abaac60dcd47b4e298a, State: Running, Role: LEADER
I20260924 23:20:12.902328 23197 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:12.902374 23202 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cce45d46304abaac60dcd47b4e298a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38809 } }
I20260924 23:20:12.902438 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:20:12.902688 23202 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5cce45d46304abaac60dcd47b4e298a. Latest consensus state: current_term: 1 leader_uuid: "d5cce45d46304abaac60dcd47b4e298a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cce45d46304abaac60dcd47b4e298a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38809 } } }
I20260924 23:20:12.902750 23202 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:12.903620 23203 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5cce45d46304abaac60dcd47b4e298a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cce45d46304abaac60dcd47b4e298a" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 38809 } } }
I20260924 23:20:12.903738 23203 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:12.903807 23208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:20:12.904414 23208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:20:12.905006 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:12.903949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41651 (local address 127.1.4.254:38809)
0924 23:20:12.903980 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:12.903981 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:12.904172 (+ 191us) server_negotiation.cc:407] Connection header received
0924 23:20:12.904222 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:12.904223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:12.904229 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:12.904269 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:12.904830 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:12.904831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:12.904832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:20:12.904836 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:12.904849 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:20:12.904853 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:12.904898 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:12.904936 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 23:20:12.904949 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260924 23:20:12.905980 23208 catalog_manager.cc:1347] Generated new cluster ID: f383ab54e9104e26b316af23da507346
I20260924 23:20:12.906028 23208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:20:12.920533 23208 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:20:12.920722 23208 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:20:12.930986 23208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5cce45d46304abaac60dcd47b4e298a: Generated new TSK 0
I20260924 23:20:12.931202 23208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:20:12.936786 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 23129
2026-09-24T23:20:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1225 ms)
[----------] 18 tests from AuthTokenIssuingTest (22337 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:20:12 dist-test-slave-s1hw krb5kdc[23228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:20:12 dist-test-slave-s1hw krb5kdc[23228](info): set up 2 sockets
Sep 24 23:20:12 dist-test-slave-s1hw krb5kdc[23228](info): commencing operation
krb5kdc: starting...
W20260924 23:20:13.895282 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:20:13 dist-test-slave-s1hw krb5kdc[23228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:20:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:20:14.014492 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:41885
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:41885,127.1.4.253:42369,127.1.4.252:38197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:20:14.094359 23241 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 23:20:14.094599 23241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:14.094623 23241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:14.094650 23241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:20:14.095854 23241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:14.095899 23241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:14.095919 23241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:14.095934 23241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:14.095950 23241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:20:14.097496 23241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:41885,127.1.4.253:42369,127.1.4.252:38197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:41885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:14.097756 23241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:14.097973 23241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:14.102689 23246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:14.102710 23247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:14.102710 23241 server_base.cc:1034] running on GCE node
W20260924 23:20:14.102950 23249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:14.103193 23241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:14.103485 23241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:14.104636 23241 hybrid_clock.cc:648] HybridClock initialized: now 1790292014104617 us; error 28 us; skew 500 ppm
Sep 24 23:20:14 dist-test-slave-s1hw krb5kdc[23228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:14.107123 23241 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:20:14.107671 23241 webserver.cc:460] Webserver started at http://127.1.4.254:43549/ using document root <none> and password file <none>
I20260924 23:20:14.107910 23241 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:14.107965 23241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:14.108106 23241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:14.109241 23241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "87c82351f8634e7c8fc5bc4f8ecd9282"
format_stamp: "Formatted at 2026-09-24 23:20:14 on dist-test-slave-s1hw"
server_key: "ccc420200f1d16162b39da8b6426ad87"
server_key_iv: "d73da0736bda38ae4296a6b96703b628"
server_key_version: "encryptionkey@0"
I20260924 23:20:14.109591 23241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "87c82351f8634e7c8fc5bc4f8ecd9282"
format_stamp: "Formatted at 2026-09-24 23:20:14 on dist-test-slave-s1hw"
server_key: "ccc420200f1d16162b39da8b6426ad87"
server_key_iv: "d73da0736bda38ae4296a6b96703b628"
server_key_version: "encryptionkey@0"
I20260924 23:20:14.111050 23241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:20:14.111891 23256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:14.112053 23241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:20:14.112146 23241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "87c82351f8634e7c8fc5bc4f8ecd9282"
format_stamp: "Formatted at 2026-09-24 23:20:14 on dist-test-slave-s1hw"
server_key: "ccc420200f1d16162b39da8b6426ad87"
server_key_iv: "d73da0736bda38ae4296a6b96703b628"
server_key_version: "encryptionkey@0"
I20260924 23:20:14.112207 23241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:14.123220 23241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:14.125384 23241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:14.125538 23241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:14.129257 23241 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:41885
I20260924 23:20:14.129293 23308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:41885 every 8 connection(s)
I20260924 23:20:14.129623 23241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:20:14.130658 23309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:14.132272 23309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:20:14 dist-test-slave-s1hw krb5kdc[23228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:20:14.139329 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 23241
I20260924 23:20:14.139472 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:20:14.139683 1043 external_mini_cluster.cc:1428] Setting key e6ee0a0a25373c3c0113f0a14e0c87ad
I20260924 23:20:14.142282 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.133077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.1.4.254:41885)
0924 23:20:14.133196 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.133201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.133232 (+ 31us) server_negotiation.cc:407] Connection header received
0924 23:20:14.133748 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.133750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.133767 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.133849 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.134393 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.135309 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.136167 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.136461 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.136572 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.138584 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:14.138591 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:14.138591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:14.138598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:14.140500 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.140945 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.140949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.140950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.141066 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.141327 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.141329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.141329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.141644 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:14.141719 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:14.141956 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 23:20:14.142083 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":93,"thread_start_us":43,"threads_started":1}
I20260924 23:20:14.142288 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.132914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:41885 (local address 127.0.0.1:36812)
0924 23:20:14.133061 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.133096 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:14.133584 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:14.133859 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:14.133862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:14.134153 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:14.134303 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.134307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.135411 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.135412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.136094 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.136096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.136552 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.136553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.136617 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.137144 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:14.137154 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:14.138500 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:14.140642 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.140649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.140651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.140853 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.141132 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.141132 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.141132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.141244 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.141711 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:14.141717 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:14.141792 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 23:20:14.141895 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 23:20:14.142153 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":108,"thread_start_us":71,"threads_started":1}
I20260924 23:20:14.143088 23309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:20:14.143463 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.143327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:42369 (local address 127.0.0.1:57476)
0924 23:20:14.143390 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.143438 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 23:20:14.143604 23257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.4.253:42369: connect: Connection refused (error 111)
I20260924 23:20:14.144035 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.143935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:42369 (local address 127.0.0.1:57482)
0924 23:20:14.143981 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.144012 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 23:20:14.144140 23309 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.253:42369: Network error: Client connection negotiation failed: client connection to 127.1.4.253:42369: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.1.4.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.253.keytab.
Entry for principal kudu/127.1.4.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.253.keytab.
I20260924 23:20:14.166198 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.253:42369
--webserver_interface=127.1.4.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.253.keytab
--principal=kudu/127.1.4.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:41885,127.1.4.253:42369,127.1.4.252:38197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:20:14.193388 23309 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } attempt: 1
I20260924 23:20:14.194020 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.193852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:42369 (local address 127.0.0.1:57496)
0924 23:20:14.193927 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.193980 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 23:20:14.194377 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.194290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:42369 (local address 127.0.0.1:57502)
0924 23:20:14.194330 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.194357 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 23:20:14.194473 23309 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.253:42369: Network error: Client connection negotiation failed: client connection to 127.1.4.253:42369: connect: Connection refused (error 111)
W20260924 23:20:14.248394 23316 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 23:20:14.248634 23316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:14.248652 23316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:14.248672 23316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:20:14.249830 23316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:14.249873 23316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:14.249886 23316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:14.249897 23316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:14.249908 23316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:20:14.251639 23316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:41885,127.1.4.253:42369,127.1.4.252:38197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.253.keytab
--principal=kudu/127.1.4.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.253:42369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.4.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:14.251895 23316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:14.252120 23316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:14.256933 23321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:14.256940 23322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:14.257026 23316 server_base.cc:1034] running on GCE node
W20260924 23:20:14.256933 23324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:14.257401 23316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:14.257699 23316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:14.258846 23316 hybrid_clock.cc:648] HybridClock initialized: now 1790292014258831 us; error 33 us; skew 500 ppm
Sep 24 23:20:14 dist-test-slave-s1hw krb5kdc[23228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:14.261548 23316 init.cc:377] Logged in from keytab as kudu/127.1.4.253@KRBTEST.COM (short username kudu)
I20260924 23:20:14.261992 23316 webserver.cc:460] Webserver started at http://127.1.4.253:43641/ using document root <none> and password file <none>
I20260924 23:20:14.262203 23316 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:14.262252 23316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:14.262377 23316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:14.262629 23309 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } attempt: 2
I20260924 23:20:14.263015 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.262875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:42369 (local address 127.0.0.1:57504)
0924 23:20:14.262938 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.262977 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 23:20:14.263317 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.263230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:42369 (local address 127.0.0.1:57512)
0924 23:20:14.263268 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.263295 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 23:20:14.263417 23309 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.253:42369: Network error: Client connection negotiation failed: client connection to 127.1.4.253:42369: connect: Connection refused (error 111)
I20260924 23:20:14.263701 23316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/data/instance:
uuid: "9396c411943e4ca1856ac86cd599cbd2"
format_stamp: "Formatted at 2026-09-24 23:20:14 on dist-test-slave-s1hw"
server_key: "bb403a2c94aae003e3008ec91dce2b73"
server_key_iv: "35b7b6c3efb646a2c4834f2623130ea7"
server_key_version: "encryptionkey@0"
I20260924 23:20:14.264026 23316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/wal/instance:
uuid: "9396c411943e4ca1856ac86cd599cbd2"
format_stamp: "Formatted at 2026-09-24 23:20:14 on dist-test-slave-s1hw"
server_key: "bb403a2c94aae003e3008ec91dce2b73"
server_key_iv: "35b7b6c3efb646a2c4834f2623130ea7"
server_key_version: "encryptionkey@0"
I20260924 23:20:14.265326 23316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:20:14.266116 23331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:14.266275 23316 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:20:14.266355 23316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/wal
uuid: "9396c411943e4ca1856ac86cd599cbd2"
format_stamp: "Formatted at 2026-09-24 23:20:14 on dist-test-slave-s1hw"
server_key: "bb403a2c94aae003e3008ec91dce2b73"
server_key_iv: "35b7b6c3efb646a2c4834f2623130ea7"
server_key_version: "encryptionkey@0"
I20260924 23:20:14.266422 23316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:14.300498 23316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:14.302567 23316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:14.302680 23316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:14.306486 23316 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.253:42369
I20260924 23:20:14.306540 23383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.253:42369 every 8 connection(s)
I20260924 23:20:14.306845 23316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/data/info.pb
I20260924 23:20:14.307832 23384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:14.309384 23384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:20:14.311884 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 23316
I20260924 23:20:14.311981 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-1/wal/instance
I20260924 23:20:14.312146 1043 external_mini_cluster.cc:1428] Setting key 916a1006be80ca29c92aa4e337e40159
Sep 24 23:20:14 dist-test-slave-s1hw krb5kdc[23228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.253@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.4.252@KRBTEST.COM; defaulting to no policy
I20260924 23:20:14.319120 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.310121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.1.4.254:41885)
0924 23:20:14.310189 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.310191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.310337 (+ 146us) server_negotiation.cc:407] Connection header received
0924 23:20:14.310842 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.310844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.310858 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.310913 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.311557 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.312099 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.313063 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.313352 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.313422 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.315966 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:14.315968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:14.315968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:14.316006 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:14.317565 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.318093 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.318095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.318095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.318154 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.318429 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.318430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.318430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.318564 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:14.318622 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:14.318932 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 23:20:14.319012 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 23:20:14.319337 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.310036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:41885 (local address 127.0.0.1:36828)
0924 23:20:14.310212 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.310261 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:14.310742 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:14.310984 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:14.310987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:14.311237 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:14.311469 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.311474 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.312175 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.312176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.312982 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.312984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.313462 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.313463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.313651 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.314144 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:14.314152 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:14.315847 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:14.317677 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.317685 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.317688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.317988 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.318226 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.318227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.318227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.318368 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.318628 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:14.318634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:14.318807 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 23:20:14.318922 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 23:20:14.319095 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260924 23:20:14.319890 23384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.252@KRBTEST.COM" created.
Sep 24 23:20:14 dist-test-slave-s1hw krb5kdc[23228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.253@KRBTEST.COM for kudu/127.1.4.253@KRBTEST.COM
I20260924 23:20:14.327919 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.320140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:42369 (local address 127.0.0.1:57526)
0924 23:20:14.320184 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.320194 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:14.320236 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:14.320439 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:14.320440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:14.320547 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:14.320655 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.320657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.321797 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.321798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.322512 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.322514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.323154 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.323155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.323213 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.323651 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:14.323656 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:14.325074 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:14.326785 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.326786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.326787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.326995 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.327197 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.327198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.327198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.327261 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.327554 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:14.327555 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:14.327604 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 23:20:14.327671 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 23:20:14.327779 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 23:20:14.327939 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.320180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57526 (local address 127.1.4.253:42369)
0924 23:20:14.320321 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.320326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.320344 (+ 18us) server_negotiation.cc:407] Connection header received
0924 23:20:14.320364 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.320365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.320376 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.320434 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.320721 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.321724 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.322633 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.323081 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.323152 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.325153 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:14.325162 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:14.325162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:14.325168 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:14.326701 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.327092 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.327095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.327095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.327144 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.327314 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.327314 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.327314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.327483 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:14.327545 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:14.327706 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 23:20:14.327838 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260924 23:20:14.328590 23384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:20:14.328917 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.328796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:38197 (local address 127.0.0.1:47086)
0924 23:20:14.328843 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.328894 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 23:20:14.329035 23332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.4.252:38197: connect: Connection refused (error 111)
I20260924 23:20:14.329432 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.329330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:38197 (local address 127.0.0.1:47088)
0924 23:20:14.329387 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.329413 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 23:20:14.329526 23384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.252:38197: Network error: Client connection negotiation failed: client connection to 127.1.4.252:38197: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.252.keytab.
Entry for principal kudu/127.1.4.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.252.keytab.
I20260924 23:20:14.336758 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.252:38197
--webserver_interface=127.1.4.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.252.keytab
--principal=kudu/127.1.4.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:41885,127.1.4.253:42369,127.1.4.252:38197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:20:14.354592 23309 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } attempt: 3
Sep 24 23:20:14 dist-test-slave-s1hw krb5kdc[23228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for kudu/127.1.4.253@KRBTEST.COM
I20260924 23:20:14.361497 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.354988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:42369 (local address 127.0.0.1:57540)
0924 23:20:14.355065 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.355076 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:14.355126 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:14.355259 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:14.355260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:14.355461 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:14.355525 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.355527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.356173 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.356174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.356693 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.356694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.357067 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.357067 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.357152 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.357557 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:14.357563 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:14.358946 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:14.360624 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.360625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.360626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.360794 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.360968 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.360969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.360969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.361014 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.361253 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:14.361254 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:14.361297 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 23:20:14.361348 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 23:20:14.361425 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 23:20:14.361543 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.355078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.1.4.253:42369)
0924 23:20:14.355150 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.355151 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.355162 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:20:14.355190 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.355191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.355203 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.355248 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.355604 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.356097 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.356767 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.356998 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.357055 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.359042 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:14.359043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:14.359044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:14.359053 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:14.360525 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.360856 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.360857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.360858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.360916 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.361066 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.361066 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.361067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.361195 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:14.361254 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:14.361380 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 23:20:14.361454 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 23:20:14.361915 23309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:20:14.362257 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.362151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:38197 (local address 127.0.0.1:47094)
0924 23:20:14.362197 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.362230 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 23:20:14.362526 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.362454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:38197 (local address 127.0.0.1:47106)
0924 23:20:14.362487 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.362510 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 23:20:14.362603 23309 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.252:38197: Network error: Client connection negotiation failed: client connection to 127.1.4.252:38197: connect: Connection refused (error 111)
I20260924 23:20:14.378767 23384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } attempt: 1
I20260924 23:20:14.379318 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.379178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:38197 (local address 127.0.0.1:47114)
0924 23:20:14.379240 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.379285 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 23:20:14.379889 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.379746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:38197 (local address 127.0.0.1:47126)
0924 23:20:14.379819 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.379857 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 23:20:14.380065 23384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.252:38197: Network error: Client connection negotiation failed: client connection to 127.1.4.252:38197: connect: Connection refused (error 111)
I20260924 23:20:14.393754 23309 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } attempt: 1
I20260924 23:20:14.394279 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.394131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:38197 (local address 127.0.0.1:47130)
0924 23:20:14.394190 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.394242 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 23:20:14.394660 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.394578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:38197 (local address 127.0.0.1:47144)
0924 23:20:14.394611 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.394639 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260924 23:20:14.394773 23309 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.252:38197: Network error: Client connection negotiation failed: client connection to 127.1.4.252:38197: connect: Connection refused (error 111)
W20260924 23:20:14.420439 23391 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 23:20:14.420694 23391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:14.420718 23391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:14.420740 23391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:20:14.421969 23391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:14.422024 23391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:14.422037 23391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:14.422049 23391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:14.422060 23391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:20:14.423705 23391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:41885,127.1.4.253:42369,127.1.4.252:38197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.252.keytab
--principal=kudu/127.1.4.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.252:38197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.4.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:14.423960 23391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:14.424190 23391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:14.429023 23396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:14.429054 23397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:14.429221 23391 server_base.cc:1034] running on GCE node
W20260924 23:20:14.429054 23399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:14.429522 23391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:14.429843 23391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:14.431010 23391 hybrid_clock.cc:648] HybridClock initialized: now 1790292014430988 us; error 34 us; skew 500 ppm
Sep 24 23:20:14 dist-test-slave-s1hw krb5kdc[23228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:14.433921 23391 init.cc:377] Logged in from keytab as kudu/127.1.4.252@KRBTEST.COM (short username kudu)
I20260924 23:20:14.434389 23391 webserver.cc:460] Webserver started at http://127.1.4.252:45019/ using document root <none> and password file <none>
I20260924 23:20:14.434602 23391 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:14.434648 23391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:14.434769 23391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:14.435901 23391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/data/instance:
uuid: "42fbf410c8da4621974afaf61007800b"
format_stamp: "Formatted at 2026-09-24 23:20:14 on dist-test-slave-s1hw"
server_key: "d97173c0a58e815b86b988a4f90ae2cc"
server_key_iv: "404ef1193e0c01b754500472063e0b02"
server_key_version: "encryptionkey@0"
I20260924 23:20:14.436224 23391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/wal/instance:
uuid: "42fbf410c8da4621974afaf61007800b"
format_stamp: "Formatted at 2026-09-24 23:20:14 on dist-test-slave-s1hw"
server_key: "d97173c0a58e815b86b988a4f90ae2cc"
server_key_iv: "404ef1193e0c01b754500472063e0b02"
server_key_version: "encryptionkey@0"
I20260924 23:20:14.437537 23391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:20:14.438308 23406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:14.438479 23391 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:20:14.438562 23391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/wal
uuid: "42fbf410c8da4621974afaf61007800b"
format_stamp: "Formatted at 2026-09-24 23:20:14 on dist-test-slave-s1hw"
server_key: "d97173c0a58e815b86b988a4f90ae2cc"
server_key_iv: "404ef1193e0c01b754500472063e0b02"
server_key_version: "encryptionkey@0"
I20260924 23:20:14.438625 23391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:14.448230 23384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } attempt: 2
I20260924 23:20:14.448722 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.448572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:38197 (local address 127.0.0.1:47150)
0924 23:20:14.448658 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.448692 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 23:20:14.449038 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.448960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:38197 (local address 127.0.0.1:47158)
0924 23:20:14.448996 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.449020 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:38197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 23:20:14.449116 23384 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.252:38197: Network error: Client connection negotiation failed: client connection to 127.1.4.252:38197: connect: Connection refused (error 111)
I20260924 23:20:14.461331 23391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:14.463464 23391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:14.463604 23391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:14.467598 23391 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.252:38197
I20260924 23:20:14.467653 23458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.252:38197 every 8 connection(s)
I20260924 23:20:14.467991 23391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/data/info.pb
I20260924 23:20:14.468953 23459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:14.469944 23309 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } attempt: 2
I20260924 23:20:14.470746 23459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:20:14.471823 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 23391
I20260924 23:20:14.471921 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291755455739-1043-0/minicluster-data/master-2/wal/instance
I20260924 23:20:14.472126 1043 external_mini_cluster.cc:1428] Setting key f35b59ea8fa4ab71ac93a28ed320c8e6
Sep 24 23:20:14 dist-test-slave-s1hw krb5kdc[23228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for kudu/127.1.4.252@KRBTEST.COM
Sep 24 23:20:14 dist-test-slave-s1hw krb5kdc[23228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.252@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
Sep 24 23:20:14 dist-test-slave-s1hw krb5kdc[23228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.252@KRBTEST.COM
I20260924 23:20:14.480219 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.470331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:38197 (local address 127.0.0.1:47174)
0924 23:20:14.470391 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.470405 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:14.470464 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:14.471575 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:14.471576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:14.471764 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:14.471820 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.471823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.473307 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.473308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.474099 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.474102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.474814 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.474815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.474872 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.475327 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:14.475332 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:14.476653 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:14.478905 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.478906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.478906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.479083 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.479427 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.479428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.479429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.479515 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.479882 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:14.479883 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:14.479928 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 23:20:14.479996 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 23:20:14.480107 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":2}
I20260924 23:20:14.480455 23462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.471775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:41885 (local address 127.0.0.1:36844)
0924 23:20:14.472410 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.472453 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:14.472547 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:14.472720 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:14.472723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:14.473039 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:14.473163 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.473168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.473863 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.473864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.474646 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.474648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.475097 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.475099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.475208 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.475617 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:14.475627 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:14.477290 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:14.478820 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.478827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.478830 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.479093 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.479260 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.479260 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.479261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.479350 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.479709 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:14.479713 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:14.479807 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 23:20:14.480129 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 23:20:14.480258 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":38,"thread_start_us":91,"threads_started":1}
I20260924 23:20:14.480696 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.472319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36844 (local address 127.1.4.254:41885)
0924 23:20:14.472370 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.472372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.472510 (+ 138us) server_negotiation.cc:407] Connection header received
0924 23:20:14.472612 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.472613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.472623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.472649 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.473256 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.473785 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.474727 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.474999 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.475050 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.477364 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:14.477365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:14.477365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:14.477372 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:14.478752 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.479157 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.479158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.479158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.479218 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.479425 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.479426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.479426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.479638 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:14.479712 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:14.479958 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 23:20:14.480041 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:20:14.480455 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.470475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.1.4.252:38197)
0924 23:20:14.470663 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.470673 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.470706 (+ 33us) server_negotiation.cc:407] Connection header received
0924 23:20:14.471315 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.471325 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.471429 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.471587 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.471940 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.473250 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.474180 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.474750 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.474843 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.476734 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:14.476741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:14.476741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:14.476748 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:14.478834 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.479285 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.479287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.479288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.479351 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.479570 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.479571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.479571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.479813 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:14.479895 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:14.480041 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 23:20:14.480214 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20260924 23:20:14.481267 23459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:20:14.482100 23463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.472460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.1.4.252:38197)
0924 23:20:14.472668 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.472670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.472707 (+ 37us) server_negotiation.cc:407] Connection header received
0924 23:20:14.472772 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.472773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.472783 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.472848 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.473466 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.474179 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.474967 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.475301 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.475382 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.478190 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:14.478191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:14.478192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:14.478198 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:14.480576 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.480940 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.480941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.480941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.480993 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.481204 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.481205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.481205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.481322 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:14.481445 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:14.481891 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 23:20:14.481956 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":191,"thread_start_us":155,"threads_started":1}
I20260924 23:20:14.482769 23309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282: Bootstrap starting.
I20260924 23:20:14.483414 23309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:14.483721 23309 log.cc:824] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:14.484848 23309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282: No bootstrap required, opened a new log
Sep 24 23:20:14 dist-test-slave-s1hw krb5kdc[23228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.252@KRBTEST.COM for kudu/127.1.4.253@KRBTEST.COM
I20260924 23:20:14.486760 23309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } }
I20260924 23:20:14.486893 23309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c82351f8634e7c8fc5bc4f8ecd9282, State: Initialized, Role: FOLLOWER
I20260924 23:20:14.486969 23309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } }
I20260924 23:20:14.487250 23466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } } }
I20260924 23:20:14.487335 23309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:14.487346 23466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:20:14.488428 23462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.481516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:42369 (local address 127.0.0.1:57552)
0924 23:20:14.481570 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.481580 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:14.481614 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:14.481778 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:14.481780 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:14.481914 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:14.481952 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.481954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.482796 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.482797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.483310 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.483312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.483779 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.483780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.483843 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.484213 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:14.484217 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:14.485558 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:14.487307 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.487308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.487309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.487534 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.487752 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.487752 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.487753 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.487817 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.488132 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:14.488133 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:14.488180 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 23:20:14.488240 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 23:20:14.488329 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
Sep 24 23:20:14 dist-test-slave-s1hw krb5kdc[23228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:20:14.488710 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.481634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.1.4.253:42369)
0924 23:20:14.481672 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.481674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.481681 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:20:14.481704 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.481705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.481714 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.481768 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.482027 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.482713 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.483415 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.483705 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.483780 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.485665 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:14.485666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:14.485666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:14.485675 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:14.487200 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.487615 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.487617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.487617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.487681 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.487908 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.487909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.487910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.488038 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:14.488104 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:14.488544 (+ 440us) server_negotiation.cc:299] Negotiation successful
0924 23:20:14.488632 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:20:14.489023 23459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } has no permanent_uuid. Determining permanent_uuid...
W20260924 23:20:14.489871 23477 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:20:14.489923 23477 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:20:14.491830 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.484214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36854 (local address 127.1.4.254:41885)
0924 23:20:14.484264 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.484266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.484275 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:20:14.484295 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.484296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.484305 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.484356 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.484913 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.485518 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.486401 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.486679 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.486737 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.489102 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:14.489103 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:14.489103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:14.489111 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:14.490582 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.490939 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.490940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.490941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.490991 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.491188 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.491189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.491189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.491307 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:14.491433 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:14.491657 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 23:20:14.491741 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 23:20:14.490689 23241 master_runner.cc:186] Error getting master registration for 127.1.4.253:42369: OK, instance_id { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" instance_seqno: 1790292014305411 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 23:20:14.493058 23241 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 24 23:20:14 dist-test-slave-s1hw krb5kdc[23228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.252@KRBTEST.COM for kudu/127.1.4.252@KRBTEST.COM
I20260924 23:20:14.497563 23462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.489261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:38197 (local address 127.0.0.1:47186)
0924 23:20:14.489326 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.489336 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:14.489367 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:14.489634 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:14.489635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:14.489996 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:14.490034 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.490036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.490853 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.490854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.491595 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.491597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.492035 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.492036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.492095 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.492461 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:14.492465 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:14.493879 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:14.496435 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.496436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.496436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.496635 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.496832 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.496833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.496834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.496890 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.497215 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:14.497216 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:14.497278 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 23:20:14.497350 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 23:20:14.497446 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 23:20:14.497643 23463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.489296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47186 (local address 127.1.4.252:38197)
0924 23:20:14.489526 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.489528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.489536 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:20:14.489554 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.489555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.489564 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.490109 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.490132 (+ 23us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.490780 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.491676 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.491967 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.493006 (+ 1039us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.493959 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:14.493960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:14.493961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:14.493968 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:14.496366 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.496706 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.496708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.496708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.496764 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.496949 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.496950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.496950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.497145 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:14.497215 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:14.497392 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 23:20:14.497511 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":210}
Sep 24 23:20:14 dist-test-slave-s1hw krb5kdc[23228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.253@KRBTEST.COM
I20260924 23:20:14.499223 23459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b: Bootstrap starting.
I20260924 23:20:14.499989 23459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:14.500270 23459 log.cc:824] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:14.500746 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.493577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.1.4.253:42369)
0924 23:20:14.493614 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.493615 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.493622 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:20:14.493641 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.493641 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.493650 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.493693 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.494395 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.495122 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.495944 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.496153 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.496209 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.498552 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:14.498553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:14.498554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:14.498561 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:14.499818 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.500104 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.500105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.500105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.500155 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.500308 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.500309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.500309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.500413 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:14.500556 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:14.500593 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 23:20:14.500662 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:20:14.501471 23459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b: No bootstrap required, opened a new log
I20260924 23:20:14.501950 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:20:14.502734 23283 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36854
I20260924 23:20:14.502867 23283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 23:20:14.502918 23283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:20:14.503515 23459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } }
I20260924 23:20:14.503670 23459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42fbf410c8da4621974afaf61007800b, State: Initialized, Role: FOLLOWER
I20260924 23:20:14.503717 23283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } }
I20260924 23:20:14.503780 23459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } }
I20260924 23:20:14.503968 23283 leader_election.cc:288] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [CANDIDATE]: Term 1 election: Requested vote from peers 9396c411943e4ca1856ac86cd599cbd2 (127.1.4.253:42369), 42fbf410c8da4621974afaf61007800b (127.1.4.252:38197)
I20260924 23:20:14.504103 23459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:14.504304 23433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "42fbf410c8da4621974afaf61007800b"
I20260924 23:20:14.504417 23433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:20:14.505657 23433 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87c82351f8634e7c8fc5bc4f8ecd9282 in term 1.
I20260924 23:20:14.505927 23257 leader_election.cc:302] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42fbf410c8da4621974afaf61007800b, 87c82351f8634e7c8fc5bc4f8ecd9282; no voters:
I20260924 23:20:14.506062 23466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:20:14.506201 23466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [term 1 LEADER]: Becoming Leader. State: Replica: 87c82351f8634e7c8fc5bc4f8ecd9282, State: Running, Role: LEADER
I20260924 23:20:14.506281 23466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } }
W20260924 23:20:14.506359 23491 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:20:14.506402 23491 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:20:14.506700 23466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87c82351f8634e7c8fc5bc4f8ecd9282. Latest consensus state: current_term: 1 leader_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } } }
I20260924 23:20:14.506713 23480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } } }
I20260924 23:20:14.506773 23466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:14.506775 23480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:20:14.507125 23499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:20:14.507323 23499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:20:14.509136 23433 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b [term 1 FOLLOWER]: Refusing update from remote peer 87c82351f8634e7c8fc5bc4f8ecd9282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:20:14.509410 23466 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:20:14.508186 23391 master_runner.cc:186] Error getting master registration for 127.1.4.253:42369: OK, instance_id { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" instance_seqno: 1790292014305411 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 23:20:14.510393 23391 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 23:20:14.510946 23480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87c82351f8634e7c8fc5bc4f8ecd9282. Latest consensus state: current_term: 1 leader_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } } }
I20260924 23:20:14.511042 23480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:20:14.512609 23466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } } }
I20260924 23:20:14.512694 23466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:14.513178 23499 catalog_manager.cc:1347] Generated new cluster ID: 46d86d3c9586448a8afc81386fd33bd4
I20260924 23:20:14.513222 23499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:20:14.519429 23480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } } }
I20260924 23:20:14.519562 23480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:20:14.519894 23501 mvcc.cc:205] Tried to move back new op lower bound from 7333036091427467264 to 7333036091418451968. Current Snapshot: MvccSnapshot[applied={T|T < 7333036091427467264}]
I20260924 23:20:14.529978 23499 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:20:14.530747 23499 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:20:14.533761 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.511610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.1.4.254:41885)
0924 23:20:14.511664 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.511666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.520180 (+ 8514us) server_negotiation.cc:407] Connection header received
0924 23:20:14.520284 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.520286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.520297 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.520326 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.521115 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.521872 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.528472 (+ 6600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.528751 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.528816 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.530562 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:14.530564 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:14.530564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:14.530571 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:14.532289 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.532677 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.532678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.532679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.532734 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.532913 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.532914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.532914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.533028 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:14.533116 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:14.533332 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 23:20:14.533429 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:20:14.540316 23384 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } attempt: 3
I20260924 23:20:14.541492 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.511379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.1.4.253:42369)
0924 23:20:14.511420 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.511422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.528704 (+ 17282us) server_negotiation.cc:407] Connection header received
0924 23:20:14.528807 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.528808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.528820 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.528849 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.529696 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.530425 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.532711 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.533025 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.533076 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.535725 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:14.535727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:14.535727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:14.535749 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:14.537687 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.540162 (+ 2475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.540164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.540164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.540241 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.540720 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.540721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.540721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.540876 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:14.540955 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:14.541330 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 23:20:14.541405 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:20:14.541793 23499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282: Generated new TSK 0
I20260924 23:20:14.542656 23499 catalog_manager.cc:1514] Initializing in-progress tserver states...
Sep 24 23:20:14 dist-test-slave-s1hw krb5kdc[23228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.253@KRBTEST.COM for kudu/127.1.4.252@KRBTEST.COM
I20260924 23:20:14.547207 23463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.512208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47202 (local address 127.1.4.252:38197)
0924 23:20:14.512249 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.512251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.534318 (+ 22067us) server_negotiation.cc:407] Connection header received
0924 23:20:14.534413 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.534414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.534426 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.534454 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.534973 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.535704 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.537313 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.537618 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.537660 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.538885 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:14.538886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:14.538886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:14.538893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:14.540469 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.545136 (+ 4667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.545137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.545138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.545193 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.545377 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.545378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.545378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.545501 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:14.545582 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:14.547017 (+ 1435us) server_negotiation.cc:299] Negotiation successful
0924 23:20:14.547109 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:20:14.549656 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.540679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:38197 (local address 127.0.0.1:47216)
0924 23:20:14.540824 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:14.540835 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:14.540871 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:14.541112 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:14.541113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:14.541263 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:14.541322 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.541324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.542739 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.542739 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.543530 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:14.543533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.544500 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.544501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:14.544554 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.544931 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:14.544935 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:14.546493 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:14.548546 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.548557 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.548558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.548760 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.549060 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:14.549061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:14.549062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:14.549124 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:14.549349 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:14.549350 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:14.549394 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 23:20:14.549455 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 23:20:14.549559 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260924 23:20:14.549657 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:14.540953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.1.4.252:38197)
0924 23:20:14.540991 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.540993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.540999 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:20:14.541016 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.541017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.541026 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.541069 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.541404 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.542118 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.543662 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:14.544427 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:14.544489 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:14.546566 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:14.546568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:14.546568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:14.546575 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:14.548392 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.548827 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.548828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.548829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.548893 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:14.549179 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:14.549180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:14.549181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:14.549298 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:14.549359 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:14.549491 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 23:20:14.549573 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:20:14.551195 23384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2: Bootstrap starting.
I20260924 23:20:14.551975 23384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:14.552266 23384 log.cc:824] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:14.553118 23384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2: No bootstrap required, opened a new log
I20260924 23:20:14.555136 23384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } }
I20260924 23:20:14.555271 23384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9396c411943e4ca1856ac86cd599cbd2, State: Initialized, Role: FOLLOWER
I20260924 23:20:14.555341 23384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } }
I20260924 23:20:14.555701 23384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:14.557716 23357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:20:14.556988 23358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9396c411943e4ca1856ac86cd599cbd2"
I20260924 23:20:14.558600 23357 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2 [term 1 FOLLOWER]: Refusing update from remote peer 87c82351f8634e7c8fc5bc4f8ecd9282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:20:14.558990 23506 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:20:14.559864 23358 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87c82351f8634e7c8fc5bc4f8ecd9282 in term 1.
I20260924 23:20:14.561199 23522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } } }
I20260924 23:20:14.561362 23522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:20:14.561441 23523 mvcc.cc:205] Tried to move back new op lower bound from 7333036091542241280 to 7333036091418451968. Current Snapshot: MvccSnapshot[applied={T|T < 7333036091427467264}]
I20260924 23:20:14.561547 23522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87c82351f8634e7c8fc5bc4f8ecd9282. Latest consensus state: current_term: 1 leader_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } } }
I20260924 23:20:14.561656 23522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:20:14.561725 23522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } } }
I20260924 23:20:14.561805 23522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 23:20:14.565246 23521 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:20:14.565304 23521 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 23:20:14.579206 23312 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:14.577536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36878 (local address 127.1.4.254:41885)
0924 23:20:14.577594 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.577596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.578510 (+ 914us) server_negotiation.cc:407] Connection header received
0924 23:20:14.578626 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.578627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.578638 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.578668 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.579129 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36878: BlockingRecv error: recv got EOF from 127.0.0.1:36878 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 23:20:14.579996 23388 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:14.577939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.1.4.253:42369)
0924 23:20:14.577976 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.577978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.579482 (+ 1504us) server_negotiation.cc:407] Connection header received
0924 23:20:14.579583 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.579584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.579595 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.579622 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.579931 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57580: BlockingRecv error: recv got EOF from 127.0.0.1:57580 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 23:20:14.580705 23461 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:14.578235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.1.4.252:38197)
0924 23:20:14.578283 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.578285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.580182 (+ 1897us) server_negotiation.cc:407] Connection header received
0924 23:20:14.580312 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.580313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.580323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.580353 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.580647 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47226: BlockingRecv error: recv got EOF from 127.0.0.1:47226 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 23:20:14.588387 23388 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:14.587305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.1.4.253:42369)
0924 23:20:14.587347 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.587349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.587753 (+ 404us) server_negotiation.cc:407] Connection header received
0924 23:20:14.587864 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.587865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.587885 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.587915 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.588327 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57592: BlockingRecv error: recv got EOF from 127.0.0.1:57592 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 23:20:14.591585 23388 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:14.590745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57598 (local address 127.1.4.253:42369)
0924 23:20:14.590778 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.590780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.590998 (+ 218us) server_negotiation.cc:407] Connection header received
0924 23:20:14.591089 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.591090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.591101 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.591128 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.591495 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57598: BlockingRecv error: recv got EOF from 127.0.0.1:57598 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 23:20:14.595496 23388 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:14.594717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.1.4.253:42369)
0924 23:20:14.594759 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.594760 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.594974 (+ 214us) server_negotiation.cc:407] Connection header received
0924 23:20:14.595067 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.595067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.595078 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.595105 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.595446 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57602: BlockingRecv error: recv got EOF from 127.0.0.1:57602 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 23:20:14.601544 23388 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:14.600782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57616 (local address 127.1.4.253:42369)
0924 23:20:14.600819 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.600821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.601033 (+ 212us) server_negotiation.cc:407] Connection header received
0924 23:20:14.601126 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.601128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.601138 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.601166 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.601489 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57616: BlockingRecv error: recv got EOF from 127.0.0.1:57616 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 23:20:14.611788 23388 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:14.610831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57628 (local address 127.1.4.253:42369)
0924 23:20:14.610871 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.610873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.611119 (+ 246us) server_negotiation.cc:407] Connection header received
0924 23:20:14.611221 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.611222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.611234 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.611263 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.611714 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57628: BlockingRecv error: recv got EOF from 127.0.0.1:57628 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 23:20:14.630093 23388 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:14.629200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57642 (local address 127.1.4.253:42369)
0924 23:20:14.629241 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.629243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.629496 (+ 253us) server_negotiation.cc:407] Connection header received
0924 23:20:14.629613 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.629614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.629626 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.629655 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.630036 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57642: BlockingRecv error: recv got EOF from 127.0.0.1:57642 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 23:20:14.664546 23388 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:14.663607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.1.4.253:42369)
0924 23:20:14.663659 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.663661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.663937 (+ 276us) server_negotiation.cc:407] Connection header received
0924 23:20:14.664038 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.664039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.664052 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.664083 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.664482 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57646: BlockingRecv error: recv got EOF from 127.0.0.1:57646 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 23:20:14.730625 23388 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:14.729908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57658 (local address 127.1.4.253:42369)
0924 23:20:14.729978 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.729980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.730062 (+ 82us) server_negotiation.cc:407] Connection header received
0924 23:20:14.730097 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.730098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.730110 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.730178 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.730577 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57658: BlockingRecv error: recv got EOF from 127.0.0.1:57658 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 23:20:14.860800 23388 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:14.859966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.1.4.253:42369)
0924 23:20:14.860072 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:14.860074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:14.860127 (+ 53us) server_negotiation.cc:407] Connection header received
0924 23:20:14.860222 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:14.860223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:14.860247 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:14.860295 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:14.860733 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57662: BlockingRecv error: recv got EOF from 127.0.0.1:57662 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 23:20:15.033339 23466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c82351f8634e7c8fc5bc4f8ecd9282" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 41885 } } peers { permanent_uuid: "9396c411943e4ca1856ac86cd599cbd2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 42369 } } peers { permanent_uuid: "42fbf410c8da4621974afaf61007800b" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 38197 } } }
I20260924 23:20:15.033469 23466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c82351f8634e7c8fc5bc4f8ecd9282 [sys.catalog]: This master's current role is: LEADER
W20260924 23:20:15.118716 23388 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:15.118082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.1.4.253:42369)
0924 23:20:15.118173 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:15.118176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:15.118186 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:20:15.118263 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:15.118265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:15.118276 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:15.118323 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:15.118653 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57664: BlockingRecv error: recv got EOF from 127.0.0.1:57664 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 23:20:15.506821 23491 catalog_manager.cc:1259] Loaded cluster ID: 46d86d3c9586448a8afc81386fd33bd4
I20260924 23:20:15.506896 23491 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b: loading cluster ID for follower catalog manager: success
I20260924 23:20:15.507750 23491 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b: acquiring CA information for follower catalog manager: success
I20260924 23:20:15.508030 23491 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 42fbf410c8da4621974afaf61007800b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 23:20:15.565745 23521 catalog_manager.cc:1259] Loaded cluster ID: 46d86d3c9586448a8afc81386fd33bd4
I20260924 23:20:15.565845 23521 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2: loading cluster ID for follower catalog manager: success
I20260924 23:20:15.566665 23521 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2: acquiring CA information for follower catalog manager: success
I20260924 23:20:15.566954 23521 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9396c411943e4ca1856ac86cd599cbd2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 23:20:15.635443 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:15.631980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57666 (local address 127.1.4.253:42369)
0924 23:20:15.632137 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:15.632139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:15.632230 (+ 91us) server_negotiation.cc:407] Connection header received
0924 23:20:15.632289 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:15.632290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:15.632371 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:15.632468 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:20:15.632621 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:15.633418 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:15.634503 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:15.634834 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:15.634907 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:15.635053 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:20:15.635178 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:20:15.635236 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:15.635302 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 23:20:15.635326 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260924 23:20:15.640956 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:15.637687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47232 (local address 127.1.4.252:38197)
0924 23:20:15.637808 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:15.637810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:15.637869 (+ 59us) server_negotiation.cc:407] Connection header received
0924 23:20:15.637916 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:15.637917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:15.637970 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:15.638071 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:20:15.638179 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:15.638967 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:15.640071 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:15.640378 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:15.640440 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:15.640617 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:20:15.640760 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:20:15.640822 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:15.640871 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 23:20:15.640890 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":53,"threads_started":1}
I20260924 23:20:15.645628 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:15.642824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.1.4.254:41885)
0924 23:20:15.643090 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:15.643091 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:15.643099 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:20:15.643158 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:15.643159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:15.643267 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:15.643341 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:20:15.643477 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:15.643979 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:15.644763 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:15.644968 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:15.645020 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:15.645190 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:20:15.645305 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:20:15.645369 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:15.645423 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 23:20:15.645447 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":50,"threads_started":1}
I20260924 23:20:15.645851 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:15.642863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.1.4.252:38197)
0924 23:20:15.642900 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:15.642901 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:15.642991 (+ 90us) server_negotiation.cc:407] Connection header received
0924 23:20:15.643043 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:15.643044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:15.643053 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:15.643101 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:20:15.643243 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:15.644001 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:15.645109 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:15.645329 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:15.645381 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:15.645530 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:20:15.645561 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:20:15.645605 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:15.645652 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 23:20:15.645671 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:20:15.645936 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:15.642716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.1.4.253:42369)
0924 23:20:15.642765 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:15.642767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:15.642872 (+ 105us) server_negotiation.cc:407] Connection header received
0924 23:20:15.642986 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:15.642987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:15.642998 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:15.643041 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:20:15.643176 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:15.644011 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:15.645138 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:15.645458 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:15.645500 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:15.645690 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:20:15.645724 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:20:15.645751 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:15.645845 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 23:20:15.645869 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:20:15.652467 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 23241
I20260924 23:20:15.657128 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 23316
I20260924 23:20:15.661366 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 23391
2026-09-24T23:20:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2725 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:20:15 dist-test-slave-s1hw krb5kdc[23611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:20:15 dist-test-slave-s1hw krb5kdc[23611](info): set up 2 sockets
Sep 24 23:20:15 dist-test-slave-s1hw krb5kdc[23611](info): commencing operation
krb5kdc: starting...
W20260924 23:20:16.615325 1043 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/test-user.keytab.
Sep 24 23:20:16 dist-test-slave-s1hw krb5kdc[23611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:20:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.4.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
Entry for principal kudu/127.1.4.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab.
I20260924 23:20:16.729800 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:40527
--webserver_interface=127.1.4.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:40527,127.1.4.253:43617,127.1.4.252:44085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:20:16.809218 23624 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 23:20:16.809432 23624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:16.809448 23624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:16.809468 23624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:20:16.810566 23624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:16.810601 23624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:16.810613 23624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:16.810624 23624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:16.810635 23624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:20:16.812064 23624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:40527,127.1.4.253:43617,127.1.4.252:44085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.254.keytab
--principal=kudu/127.1.4.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.254:40527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.4.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:16.812252 23624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:16.812425 23624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:16.817199 23632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:16.817291 23630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:16.817238 23629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:16.817296 23624 server_base.cc:1034] running on GCE node
I20260924 23:20:16.817655 23624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:16.817942 23624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:16.819131 23624 hybrid_clock.cc:648] HybridClock initialized: now 1790292016819112 us; error 32 us; skew 500 ppm
Sep 24 23:20:16 dist-test-slave-s1hw krb5kdc[23611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:16.821914 23624 init.cc:377] Logged in from keytab as kudu/127.1.4.254@KRBTEST.COM (short username kudu)
I20260924 23:20:16.822371 23624 webserver.cc:460] Webserver started at http://127.1.4.254:39083/ using document root <none> and password file <none>
I20260924 23:20:16.822589 23624 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:16.822650 23624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:16.822772 23624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:16.823894 23624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/data/instance:
uuid: "18fe3620a8df4ceb8fd1bc0b52771618"
format_stamp: "Formatted at 2026-09-24 23:20:16 on dist-test-slave-s1hw"
server_key: "c1287d45e875aceb17734103ddbda9e0"
server_key_iv: "22ea079d4fa69729fc50584f71fef0cb"
server_key_version: "encryptionkey@0"
I20260924 23:20:16.824230 23624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/wal/instance:
uuid: "18fe3620a8df4ceb8fd1bc0b52771618"
format_stamp: "Formatted at 2026-09-24 23:20:16 on dist-test-slave-s1hw"
server_key: "c1287d45e875aceb17734103ddbda9e0"
server_key_iv: "22ea079d4fa69729fc50584f71fef0cb"
server_key_version: "encryptionkey@0"
I20260924 23:20:16.825543 23624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:20:16.826320 23639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:16.826509 23624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:20:16.826599 23624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/wal
uuid: "18fe3620a8df4ceb8fd1bc0b52771618"
format_stamp: "Formatted at 2026-09-24 23:20:16 on dist-test-slave-s1hw"
server_key: "c1287d45e875aceb17734103ddbda9e0"
server_key_iv: "22ea079d4fa69729fc50584f71fef0cb"
server_key_version: "encryptionkey@0"
I20260924 23:20:16.826675 23624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:16.834883 23624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:16.837029 23624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:16.837179 23624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:16.841034 23624 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.254:40527
I20260924 23:20:16.841086 23691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.254:40527 every 8 connection(s)
I20260924 23:20:16.841418 23624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/data/info.pb
I20260924 23:20:16.842360 23692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:16.844092 23692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:20:16.844416 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 23624
I20260924 23:20:16.844506 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-0/wal/instance
I20260924 23:20:16.844682 1043 external_mini_cluster.cc:1428] Setting key eb02576fc25f86c13d596b29f79783ca
Sep 24 23:20:16 dist-test-slave-s1hw krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.4.253@KRBTEST.COM; defaulting to no policy
I20260924 23:20:16.854192 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:16.844828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40527 (local address 127.0.0.1:41834)
0924 23:20:16.844992 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:16.845035 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:16.845714 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:16.846056 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:16.846059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:16.846400 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:16.846634 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:16.846641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:16.847760 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:16.847761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:16.848790 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:16.848792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:16.849275 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:16.849276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:16.849332 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:16.849806 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:16.849814 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:16.851248 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:16.852920 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:16.852926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:16.852927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:16.853105 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:16.853307 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:16.853308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:16.853308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:16.853379 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:16.853753 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:16.853757 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:16.853814 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 23:20:16.853884 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 23:20:16.854062 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":62,"thread_start_us":78,"threads_started":1}
I20260924 23:20:16.854218 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:16.844862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41834 (local address 127.1.4.254:40527)
0924 23:20:16.845161 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:16.845166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:16.845202 (+ 36us) server_negotiation.cc:407] Connection header received
0924 23:20:16.845919 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:16.845922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:16.845945 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:16.846030 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:16.846768 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:16.847644 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:16.848870 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:16.849177 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:16.849266 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:16.851333 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:16.851340 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:16.851340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:16.851346 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:16.852853 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:16.853165 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:16.853167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:16.853168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:16.853259 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:16.853456 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:16.853457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:16.853457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:16.853686 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:16.853758 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:16.853908 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 23:20:16.854001 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":83,"threads_started":1}
I20260924 23:20:16.854924 23692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.253@KRBTEST.COM" created.
I20260924 23:20:16.855279 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:16.855162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:43617 (local address 127.0.0.1:42530)
0924 23:20:16.855218 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:16.855260 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:43617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 23:20:16.855436 23642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.4.253:43617: connect: Connection refused (error 111)
I20260924 23:20:16.855841 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:16.855759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:43617 (local address 127.0.0.1:42544)
0924 23:20:16.855797 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:16.855821 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:43617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 23:20:16.855931 23692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.253:43617: Network error: Client connection negotiation failed: client connection to 127.1.4.253:43617: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.253.keytab.
Entry for principal kudu/127.1.4.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.253.keytab.
I20260924 23:20:16.870571 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.253:43617
--webserver_interface=127.1.4.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.253.keytab
--principal=kudu/127.1.4.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:40527,127.1.4.253:43617,127.1.4.252:44085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:20:16.905164 23692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } attempt: 1
I20260924 23:20:16.905632 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:16.905512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:43617 (local address 127.0.0.1:42560)
0924 23:20:16.905566 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:16.905601 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:43617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 23:20:16.905943 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:16.905864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:43617 (local address 127.0.0.1:42562)
0924 23:20:16.905902 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:16.905925 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:43617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 23:20:16.906018 23692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.253:43617: Network error: Client connection negotiation failed: client connection to 127.1.4.253:43617: connect: Connection refused (error 111)
W20260924 23:20:16.950348 23699 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 23:20:16.950567 23699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:16.950583 23699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:16.950603 23699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:20:16.951789 23699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:16.951833 23699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:16.951845 23699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:16.951856 23699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:16.951866 23699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:20:16.953342 23699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:40527,127.1.4.253:43617,127.1.4.252:44085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.253.keytab
--principal=kudu/127.1.4.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.253:43617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.4.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:16.953574 23699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:16.953791 23699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:16.958483 23707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:16.958491 23704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:16.958547 23705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:16.958853 23699 server_base.cc:1034] running on GCE node
I20260924 23:20:16.959043 23699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:16.959283 23699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:16.960449 23699 hybrid_clock.cc:648] HybridClock initialized: now 1790292016960443 us; error 42 us; skew 500 ppm
Sep 24 23:20:16 dist-test-slave-s1hw krb5kdc[23611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:16.963030 23699 init.cc:377] Logged in from keytab as kudu/127.1.4.253@KRBTEST.COM (short username kudu)
I20260924 23:20:16.963475 23699 webserver.cc:460] Webserver started at http://127.1.4.253:43589/ using document root <none> and password file <none>
I20260924 23:20:16.963663 23699 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:16.963701 23699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:16.963793 23699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:16.964828 23699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/data/instance:
uuid: "19818426417c4b7aae19263da66c76a2"
format_stamp: "Formatted at 2026-09-24 23:20:16 on dist-test-slave-s1hw"
server_key: "09eeabb7f3ee1d2cfa5e12bffb128f6c"
server_key_iv: "4d0b37b125f9ae86f32afa1a985483f7"
server_key_version: "encryptionkey@0"
I20260924 23:20:16.965119 23699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/wal/instance:
uuid: "19818426417c4b7aae19263da66c76a2"
format_stamp: "Formatted at 2026-09-24 23:20:16 on dist-test-slave-s1hw"
server_key: "09eeabb7f3ee1d2cfa5e12bffb128f6c"
server_key_iv: "4d0b37b125f9ae86f32afa1a985483f7"
server_key_version: "encryptionkey@0"
I20260924 23:20:16.966305 23699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:20:16.967051 23714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:16.967228 23699 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:20:16.967324 23699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/wal
uuid: "19818426417c4b7aae19263da66c76a2"
format_stamp: "Formatted at 2026-09-24 23:20:16 on dist-test-slave-s1hw"
server_key: "09eeabb7f3ee1d2cfa5e12bffb128f6c"
server_key_iv: "4d0b37b125f9ae86f32afa1a985483f7"
server_key_version: "encryptionkey@0"
I20260924 23:20:16.967411 23699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:16.974151 23692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } attempt: 2
I20260924 23:20:16.974561 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:16.974446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:43617 (local address 127.0.0.1:42566)
0924 23:20:16.974497 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:16.974533 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:43617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 23:20:16.974857 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:16.974783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:43617 (local address 127.0.0.1:42582)
0924 23:20:16.974816 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:16.974839 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.253:43617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 23:20:16.974951 23692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.253:43617: Network error: Client connection negotiation failed: client connection to 127.1.4.253:43617: connect: Connection refused (error 111)
I20260924 23:20:16.985059 23699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:16.987131 23699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:16.987244 23699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:16.990957 23699 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.253:43617
I20260924 23:20:16.990988 23766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.253:43617 every 8 connection(s)
I20260924 23:20:16.991333 23699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/data/info.pb
I20260924 23:20:16.992326 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:16.993868 23767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:20:16.995052 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 23699
I20260924 23:20:16.995139 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-1/wal/instance
I20260924 23:20:16.995306 1043 external_mini_cluster.cc:1428] Setting key 23c4819dd9c43706d0743895d138a546
Sep 24 23:20:16 dist-test-slave-s1hw krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.253@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.4.252@KRBTEST.COM; defaulting to no policy
I20260924 23:20:17.003611 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:16.994569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41846 (local address 127.1.4.254:40527)
0924 23:20:16.994641 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:16.994644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:16.994874 (+ 230us) server_negotiation.cc:407] Connection header received
0924 23:20:16.995454 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:16.995459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:16.995475 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:16.995527 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:16.996152 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:16.996857 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:16.997856 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:16.998148 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:16.998220 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.000604 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:17.000606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:17.000606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:17.000613 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:17.002260 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.002634 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.002635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.002636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.002693 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.002920 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.002921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.002921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.003102 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:17.003170 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:17.003469 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 23:20:17.003542 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 23:20:17.003861 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:16.994512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40527 (local address 127.0.0.1:41846)
0924 23:20:16.994745 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:16.994784 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:16.995335 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:16.995599 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:16.995602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:16.995846 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:16.996044 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:16.996048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:16.996970 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:16.996971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:16.997776 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:16.997778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:16.998240 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:16.998241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:16.998349 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:16.998834 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:16.998843 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:17.000456 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:17.002332 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:17.002339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:17.002341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:17.002549 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:17.002753 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:17.002753 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:17.002754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:17.002850 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:17.003164 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:17.003169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:17.003293 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 23:20:17.003406 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 23:20:17.003545 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":122,"threads_started":1}
I20260924 23:20:17.004520 23767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.4.252@KRBTEST.COM" created.
Sep 24 23:20:17 dist-test-slave-s1hw krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.253@KRBTEST.COM for kudu/127.1.4.253@KRBTEST.COM
I20260924 23:20:17.012077 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.004818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:43617 (local address 127.0.0.1:42592)
0924 23:20:17.004857 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.004868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:17.004910 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:17.005163 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:17.005164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:17.005273 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:17.005326 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:17.005327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.006505 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.006506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:17.007002 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:17.007004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.007473 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.007474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:17.007533 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.007879 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:17.007884 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:17.009109 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:17.011002 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:17.011003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:17.011003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:17.011210 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:17.011415 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:17.011416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:17.011416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:17.011480 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:17.011749 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:17.011750 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:17.011798 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 23:20:17.011864 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 23:20:17.011965 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 23:20:17.012162 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.004860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.1.4.253:43617)
0924 23:20:17.005015 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.005020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.005041 (+ 21us) server_negotiation.cc:407] Connection header received
0924 23:20:17.005061 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.005063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.005076 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.005150 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.005388 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.006435 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.007069 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.007418 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.007478 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.009196 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:17.009205 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:17.009206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:17.009212 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:17.010913 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.011282 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.011285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.011285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.011335 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.011532 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.011532 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.011533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.011685 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:17.011750 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:17.011898 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 23:20:17.012026 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260924 23:20:17.012754 23767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:20:17.013099 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.012969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:44085 (local address 127.0.0.1:54784)
0924 23:20:17.013017 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.013072 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 23:20:17.013236 23716 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.4.252:44085: connect: Connection refused (error 111)
I20260924 23:20:17.013643 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.013552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:44085 (local address 127.0.0.1:54786)
0924 23:20:17.013595 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.013624 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 23:20:17.013748 23767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.252:44085: Network error: Client connection negotiation failed: client connection to 127.1.4.252:44085: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.4.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.252.keytab.
Entry for principal kudu/127.1.4.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.252.keytab.
I20260924 23:20:17.022955 1043 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEEvosl/build/release/bin/kudu
/tmp/dist-test-taskEEvosl/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.252:44085
--webserver_interface=127.1.4.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.252.keytab
--principal=kudu/127.1.4.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.4.212:40047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.4.254:40527,127.1.4.253:43617,127.1.4.252:44085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:20:17.062974 23767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } attempt: 1
I20260924 23:20:17.063619 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.063461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:44085 (local address 127.0.0.1:54792)
0924 23:20:17.063537 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.063586 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 23:20:17.064028 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.063883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:44085 (local address 127.0.0.1:54804)
0924 23:20:17.063952 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.063996 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 23:20:17.064152 23767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.252:44085: Network error: Client connection negotiation failed: client connection to 127.1.4.252:44085: connect: Connection refused (error 111)
I20260924 23:20:17.066135 23692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } attempt: 3
Sep 24 23:20:17 dist-test-slave-s1hw krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for kudu/127.1.4.253@KRBTEST.COM
I20260924 23:20:17.073702 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.066613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.1.4.253:43617)
0924 23:20:17.066665 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.066667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.066677 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:20:17.066705 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.066707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.066716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.066772 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.067148 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.067743 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.068799 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.069018 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.069079 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.071155 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:17.071156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:17.071157 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:17.071164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:17.072577 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.072936 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.072938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.072938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.073000 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.073205 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.073206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.073206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.073370 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:17.073437 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:17.073533 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 23:20:17.073610 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 23:20:17.073714 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.066478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:43617 (local address 127.0.0.1:42606)
0924 23:20:17.066543 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.066556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:17.066611 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:17.066787 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:17.066788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:17.066967 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:17.067053 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:17.067055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.067830 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.067831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:17.068703 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:17.068706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.069087 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.069088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:17.069162 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.069610 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:17.069615 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:17.071058 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:17.072665 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:17.072667 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:17.072667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:17.072868 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:17.073081 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:17.073082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:17.073082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:17.073147 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:17.073425 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:17.073425 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:17.073464 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 23:20:17.073508 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:20:17.073649 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 23:20:17.074263 23692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:20:17.074638 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.074498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:44085 (local address 127.0.0.1:54816)
0924 23:20:17.074536 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.074618 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 23:20:17.074927 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.074852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:44085 (local address 127.0.0.1:54822)
0924 23:20:17.074888 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.074909 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 23:20:17.075009 23692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.252:44085: Network error: Client connection negotiation failed: client connection to 127.1.4.252:44085: connect: Connection refused (error 111)
W20260924 23:20:17.105532 23774 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 23:20:17.105772 23774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:20:17.105786 23774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:20:17.105806 23774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 23:20:17.106187 23692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } attempt: 1
I20260924 23:20:17.106731 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.106583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:44085 (local address 127.0.0.1:54836)
0924 23:20:17.106650 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.106697 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 23:20:17.107178 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.107022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:44085 (local address 127.0.0.1:54838)
0924 23:20:17.107113 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.107152 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260924 23:20:17.107177 23774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:20:17.107223 23774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:20:17.107242 23774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:20:17.107259 23774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:20:17.107282 23774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 23:20:17.107316 23692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.252:44085: Network error: Client connection negotiation failed: client connection to 127.1.4.252:44085: connect: Connection refused (error 111)
I20260924 23:20:17.108861 23774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.4.212:40047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.4.254:40527,127.1.4.253:43617,127.1.4.252:44085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/krb5kdc/kudu_127.1.4.252.keytab
--principal=kudu/127.1.4.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.4.252:44085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.4.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:20:17.109076 23774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:20:17.109256 23774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:20:17.113926 23780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:20:17.113909 23779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:17.113996 23774 server_base.cc:1034] running on GCE node
W20260924 23:20:17.113902 23782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:20:17.114403 23774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:20:17.114679 23774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:20:17.115831 23774 hybrid_clock.cc:648] HybridClock initialized: now 1790292017115817 us; error 29 us; skew 500 ppm
Sep 24 23:20:17 dist-test-slave-s1hw krb5kdc[23611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:20:17.118199 23774 init.cc:377] Logged in from keytab as kudu/127.1.4.252@KRBTEST.COM (short username kudu)
I20260924 23:20:17.118628 23774 webserver.cc:460] Webserver started at http://127.1.4.252:33667/ using document root <none> and password file <none>
I20260924 23:20:17.118820 23774 fs_manager.cc:359] Metadata directory not provided
I20260924 23:20:17.118863 23774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:20:17.118983 23774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:20:17.120095 23774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/data/instance:
uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a"
format_stamp: "Formatted at 2026-09-24 23:20:17 on dist-test-slave-s1hw"
server_key: "fc1ca5455332fdd43e8ad06aa16130a4"
server_key_iv: "8a3cdf20493af20eb1d4422b0af2bb36"
server_key_version: "encryptionkey@0"
I20260924 23:20:17.120427 23774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/wal/instance:
uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a"
format_stamp: "Formatted at 2026-09-24 23:20:17 on dist-test-slave-s1hw"
server_key: "fc1ca5455332fdd43e8ad06aa16130a4"
server_key_iv: "8a3cdf20493af20eb1d4422b0af2bb36"
server_key_version: "encryptionkey@0"
I20260924 23:20:17.121790 23774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:20:17.122679 23789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:20:17.122898 23774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:20:17.122989 23774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/data,/tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/wal
uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a"
format_stamp: "Formatted at 2026-09-24 23:20:17 on dist-test-slave-s1hw"
server_key: "fc1ca5455332fdd43e8ad06aa16130a4"
server_key_iv: "8a3cdf20493af20eb1d4422b0af2bb36"
server_key_version: "encryptionkey@0"
I20260924 23:20:17.123052 23774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:20:17.132326 23767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } attempt: 2
I20260924 23:20:17.132885 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.132739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:44085 (local address 127.0.0.1:54846)
0924 23:20:17.132799 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.132848 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 23:20:17.133263 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.133190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:44085 (local address 127.0.0.1:54862)
0924 23:20:17.133225 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.133247 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.4.252:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 23:20:17.133378 23767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.4.252:44085: Network error: Client connection negotiation failed: client connection to 127.1.4.252:44085: connect: Connection refused (error 111)
I20260924 23:20:17.141201 23774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:20:17.143344 23774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:20:17.143553 23774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:20:17.147264 23774 rpc_server.cc:307] RPC server started. Bound to: 127.1.4.252:44085
I20260924 23:20:17.147311 23841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.4.252:44085 every 8 connection(s)
I20260924 23:20:17.147678 23774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/data/info.pb
I20260924 23:20:17.148629 23842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:20:17.150210 23842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:20:17 dist-test-slave-s1hw krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.252@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:20:17.157749 1043 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEEvosl/build/release/bin/kudu as pid 23774
I20260924 23:20:17.157881 1043 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEEvosl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291755455739-1043-0/minicluster-data/master-2/wal/instance
I20260924 23:20:17.158085 1043 external_mini_cluster.cc:1428] Setting key d6368f6f7918d7fe14a0fa408b4b1a8e
I20260924 23:20:17.158967 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.150922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.1.4.254:40527)
0924 23:20:17.150975 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.150978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.151193 (+ 215us) server_negotiation.cc:407] Connection header received
0924 23:20:17.151733 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.151734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.151747 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.151800 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.152403 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.152873 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.153838 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.154025 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.154080 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.156226 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:17.156228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:17.156228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:17.156237 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:17.157402 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.157863 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.157864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.157864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.157944 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.158203 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.158204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.158205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.158332 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:17.158395 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:17.158790 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 23:20:17.158876 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:20:17.159144 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.150876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.254:40527 (local address 127.0.0.1:41850)
0924 23:20:17.151051 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.151087 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:17.151618 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:17.151876 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:17.151880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:17.152128 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:17.152319 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:17.152323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.152949 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.152949 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:17.153754 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:17.153756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.154111 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.154113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:17.154230 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.154735 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:17.154744 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:17.156139 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:17.157475 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:17.157482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:17.157484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:17.157773 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:17.158028 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:17.158029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:17.158030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:17.158142 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:17.158416 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:17.158419 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:17.158654 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 23:20:17.158750 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 23:20:17.158927 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":4,"thread_start_us":86,"threads_started":1}
I20260924 23:20:17.159708 23842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:20:17 dist-test-slave-s1hw krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.252@KRBTEST.COM
Sep 24 23:20:17 dist-test-slave-s1hw krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.252@KRBTEST.COM for kudu/127.1.4.253@KRBTEST.COM
I20260924 23:20:17.167260 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.158417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54870 (local address 127.1.4.252:44085)
0924 23:20:17.158705 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.158710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.158723 (+ 13us) server_negotiation.cc:407] Connection header received
0924 23:20:17.158739 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.158740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.158751 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.158806 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.159480 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.160419 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.161429 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.161859 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.161931 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.164258 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:17.164266 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:17.164267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:17.164273 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:17.166084 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.166469 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.166472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.166473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.166530 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.166701 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.166702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.166702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.166868 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:17.167012 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:17.167077 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 23:20:17.167147 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":78,"threads_started":1}
I20260924 23:20:17.167312 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.160079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.1.4.253:43617)
0924 23:20:17.160125 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.160127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.160135 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:20:17.160159 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.160160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.160169 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.160215 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.160485 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.161266 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.162389 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.162693 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.162768 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.164726 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:17.164727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:17.164728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:17.164735 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:17.166216 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.166601 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.166602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.166603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.166656 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.166833 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.166834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.166834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.166955 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:17.167013 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:17.167170 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 23:20:17.167235 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:20:17.167425 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.159995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.253:43617 (local address 127.0.0.1:42620)
0924 23:20:17.160041 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.160052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:17.160091 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:17.160225 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:17.160226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:17.160360 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:17.160415 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:17.160417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.161341 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.161341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:17.162089 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:17.162091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.162781 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.162782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:17.162838 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.163195 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:17.163198 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:17.164433 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:17.166291 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:17.166292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:17.166292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:17.166471 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:17.166714 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:17.166715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:17.166715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:17.166780 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:17.167030 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:17.167030 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:17.167070 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 23:20:17.167134 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 23:20:17.167298 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 23:20:17.167819 23842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:20:17 dist-test-slave-s1hw krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.252@KRBTEST.COM for kudu/127.1.4.252@KRBTEST.COM
Sep 24 23:20:17 dist-test-slave-s1hw krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.254@KRBTEST.COM
I20260924 23:20:17.174013 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.168061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:44085 (local address 127.0.0.1:54882)
0924 23:20:17.168105 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.168114 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:17.168152 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:17.168251 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:17.168251 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:17.168364 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:17.168392 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:17.168393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.168987 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.168988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:17.169462 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:17.169463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.169768 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.169769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:17.169812 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.170183 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:17.170188 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:17.171444 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:17.172930 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:17.172931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:17.172931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:17.173167 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:17.173348 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:17.173349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:17.173349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:17.173395 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:17.173637 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:17.173638 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:17.173686 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 23:20:17.173772 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 23:20:17.173863 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 23:20:17.174050 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.168090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.1.4.252:44085)
0924 23:20:17.168120 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.168122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.168142 (+ 20us) server_negotiation.cc:407] Connection header received
0924 23:20:17.168200 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.168201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.168209 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.168249 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.168459 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.168924 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.169525 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.169716 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.169767 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.171517 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:17.171518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:17.171518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:17.171524 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:17.172847 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.173233 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.173235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.173235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.173288 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.173446 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.173446 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.173446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.173565 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:17.173620 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:17.173790 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 23:20:17.173915 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":16}
I20260924 23:20:17.174655 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.168064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41864 (local address 127.1.4.254:40527)
0924 23:20:17.168104 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.168106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.168114 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:20:17.168136 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.168137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.168146 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.168198 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.168630 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.169119 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.169832 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.170031 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.170082 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.172099 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:17.172100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:17.172101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:17.172108 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:17.173482 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.173885 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.173887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.173887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.173940 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.174188 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.174188 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.174189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.174286 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:17.174446 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:17.174481 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 23:20:17.174571 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:20:17.175779 23842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a: Bootstrap starting.
I20260924 23:20:17.176558 23842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:17.176828 23842 log.cc:824] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:17.177500 23842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a: No bootstrap required, opened a new log
I20260924 23:20:17.179061 23842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } }
I20260924 23:20:17.179224 23842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e5045c5a64e8dbcbfdc948fdc5e7a, State: Initialized, Role: FOLLOWER
I20260924 23:20:17.179301 23842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } }
Sep 24 23:20:17 dist-test-slave-s1hw krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.4.253@KRBTEST.COM
I20260924 23:20:17.179661 23842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:17.179701 23848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } } }
I20260924 23:20:17.179798 23848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a [sys.catalog]: This master's current role is: FOLLOWER
W20260924 23:20:17.181324 23859 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:20:17.181373 23859 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:20:17.182459 23692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } attempt: 2
I20260924 23:20:17.183040 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.175495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42628 (local address 127.1.4.253:43617)
0924 23:20:17.175663 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.175665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.175671 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:20:17.175689 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.175690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.175697 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.175741 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.176190 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.176783 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.177446 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.177736 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.177797 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.180088 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:17.180089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:17.180090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:17.180096 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:17.181655 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.181993 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.181994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.181995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.182064 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.182238 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.182239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.182239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.182360 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:17.182515 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:17.182872 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 23:20:17.182940 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260924 23:20:17.181653 23774 master_runner.cc:186] Error getting master registration for 127.1.4.254:40527: OK, instance_id { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" instance_seqno: 1790292016839883 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 23:20:17.184285 23774 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 23:20:17.185019 1043 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:20:17.185817 23666 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "18fe3620a8df4ceb8fd1bc0b52771618"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41864
Sep 24 23:20:17 dist-test-slave-s1hw krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.254@KRBTEST.COM for kudu/127.1.4.252@KRBTEST.COM
I20260924 23:20:17.189940 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.182687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:44085 (local address 127.0.0.1:54894)
0924 23:20:17.182767 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.182778 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:17.182815 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:17.183038 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:17.183039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:17.183212 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:17.183255 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:17.183257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.184148 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.184150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:17.184833 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:17.184835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.185233 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.185234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:17.185296 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.185785 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:17.185789 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:17.187195 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:17.188880 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:17.188881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:17.188881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:17.189093 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:17.189284 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:17.189285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:17.189285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:17.189347 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:17.189646 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:17.189647 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:17.189693 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 23:20:17.189750 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 23:20:17.189841 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 23:20:17.189951 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.182883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54894 (local address 127.1.4.252:44085)
0924 23:20:17.182920 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.182921 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.182928 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:20:17.182946 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.182946 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.182955 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.182998 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.183340 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.184053 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.184898 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.185149 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.185202 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.187279 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:17.187281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:17.187281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:17.187289 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:17.188803 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.189162 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.189163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.189163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.189223 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.189398 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.189399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.189399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.189573 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:17.189645 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:17.189787 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 23:20:17.189865 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:20:17.191960 23692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618: Bootstrap starting.
I20260924 23:20:17.192704 23692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:17.193004 23692 log.cc:824] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:17.193852 23692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618: No bootstrap required, opened a new log
I20260924 23:20:17.195230 23692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } }
I20260924 23:20:17.195407 23692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18fe3620a8df4ceb8fd1bc0b52771618, State: Initialized, Role: FOLLOWER
I20260924 23:20:17.195498 23692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } }
I20260924 23:20:17.195839 23692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:17.195861 23666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 23:20:17.195955 23666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:20:17.195818 23862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } } }
I20260924 23:20:17.196143 23862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:20:17.196748 23666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } }
I20260924 23:20:17.197315 23816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a"
I20260924 23:20:17.197451 23816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:20:17.198370 23816 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18fe3620a8df4ceb8fd1bc0b52771618 in term 1.
I20260924 23:20:17.198619 23641 leader_election.cc:302] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137e5045c5a64e8dbcbfdc948fdc5e7a, 18fe3620a8df4ceb8fd1bc0b52771618; no voters:
I20260924 23:20:17.198753 23862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:20:17.198884 23862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [term 1 LEADER]: Becoming Leader. State: Replica: 18fe3620a8df4ceb8fd1bc0b52771618, State: Running, Role: LEADER
I20260924 23:20:17.198972 23862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } }
W20260924 23:20:17.199105 23873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:20:17.199196 23873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:20:17.199402 23666 leader_election.cc:288] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [CANDIDATE]: Term 1 election: Requested vote from peers 19818426417c4b7aae19263da66c76a2 (127.1.4.253:43617), 137e5045c5a64e8dbcbfdc948fdc5e7a (127.1.4.252:44085)
I20260924 23:20:17.199510 23875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18fe3620a8df4ceb8fd1bc0b52771618. Latest consensus state: current_term: 1 leader_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } } }
I20260924 23:20:17.199610 23875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:17.198328 23624 master_runner.cc:186] Error getting master registration for 127.1.4.253:43617: OK, instance_id { permanent_uuid: "19818426417c4b7aae19263da66c76a2" instance_seqno: 1790292016989885 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 23:20:17.200567 23624 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 23:20:17.207489 23877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:20:17.207743 23877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:20:17.210659 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.201942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.1.4.254:40527)
0924 23:20:17.201976 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.201978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.202235 (+ 257us) server_negotiation.cc:407] Connection header received
0924 23:20:17.202323 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.202324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.202333 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.202362 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.203111 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.204023 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.205051 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.205338 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.205379 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.207247 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:17.207248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:17.207248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:17.207256 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:17.209131 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.209751 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.209752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.209753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.209816 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.210043 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.210044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.210044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.210174 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:17.210270 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:17.210477 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 23:20:17.210568 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:20:17.211958 23816 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a [term 1 FOLLOWER]: Refusing update from remote peer 18fe3620a8df4ceb8fd1bc0b52771618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:20:17.212252 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.201705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54898 (local address 127.1.4.252:44085)
0924 23:20:17.201745 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.201747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.202131 (+ 384us) server_negotiation.cc:407] Connection header received
0924 23:20:17.202163 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.202164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.202196 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.202279 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.203402 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.204474 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.206017 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.206459 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.206522 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.208500 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:17.208501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:17.208501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:17.208509 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:17.210717 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.211320 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.211321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.211321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.211442 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.211677 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.211678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.211679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.211811 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:17.211897 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:17.212093 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 23:20:17.212169 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:20:17.212387 23875 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:20:17.213800 23848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18fe3620a8df4ceb8fd1bc0b52771618. Latest consensus state: current_term: 1 leader_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } } }
I20260924 23:20:17.213877 23848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:20:17.214941 23862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } } }
I20260924 23:20:17.215027 23862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:17.215085 23848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } } }
I20260924 23:20:17.215153 23848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:20:17.215137 23884 mvcc.cc:205] Tried to move back new op lower bound from 7333036102497259520 to 7333036102447628288. Current Snapshot: MvccSnapshot[applied={T|T < 7333036102497259520}]
I20260924 23:20:17.215574 23877 catalog_manager.cc:1347] Generated new cluster ID: 5e599c4648fc47ddac4fdb769b5e2c35
I20260924 23:20:17.215672 23877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:20:17.220057 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.211657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.1.4.253:43617)
0924 23:20:17.211700 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.211702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.211709 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:20:17.211732 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.211733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.211742 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.211789 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.212474 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.213144 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.214207 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.214588 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.214631 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.216761 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:17.216762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:17.216762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:17.216770 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:17.218429 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.218957 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.218958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.218958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.219026 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.219301 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.219302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.219303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.219474 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:17.219571 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:17.219873 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 23:20:17.219957 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:20:17.224473 23767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } attempt: 3
Sep 24 23:20:17 dist-test-slave-s1hw krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790292016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.4.253@KRBTEST.COM for kudu/127.1.4.252@KRBTEST.COM
I20260924 23:20:17.233466 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.224767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.4.252:44085 (local address 127.0.0.1:54902)
0924 23:20:17.224808 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 23:20:17.224830 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 23:20:17.224877 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:20:17.225058 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:20:17.225059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:20:17.225217 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:20:17.225262 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:17.225264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.226796 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.226797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:17.227658 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:20:17.227660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.228191 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.228192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:20:17.228263 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.228781 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:20:17.228787 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:20:17.230315 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:20:17.232305 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:17.232306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:17.232306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:17.232543 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:17.232769 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:20:17.232770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:20:17.232770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:20:17.232838 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:20:17.233143 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:20:17.233144 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:20:17.233200 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 23:20:17.233257 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 23:20:17.233361 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 23:20:17.234449 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:17.224884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54902 (local address 127.1.4.252:44085)
0924 23:20:17.224921 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.224923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.224931 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:20:17.224961 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.224961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.224985 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.225041 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.225348 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.226111 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.227763 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:17.228069 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:17.228125 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:17.230416 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:20:17.230418 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:20:17.230418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:20:17.230425 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:20:17.232192 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.232628 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.232629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.232630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.232696 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:20:17.232905 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:20:17.232906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:20:17.232907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:20:17.233050 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:20:17.233106 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:17.234278 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0924 23:20:17.234358 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:20:17.236390 23767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2: Bootstrap starting.
I20260924 23:20:17.237152 23767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2: Neither blocks nor log segments found. Creating new log.
I20260924 23:20:17.237447 23767 log.cc:824] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2: Log is configured to *not* fsync() on all Append() calls
I20260924 23:20:17.238409 23767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2: No bootstrap required, opened a new log
I20260924 23:20:17.241197 23767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } }
I20260924 23:20:17.241420 23767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19818426417c4b7aae19263da66c76a2, State: Initialized, Role: FOLLOWER
I20260924 23:20:17.241545 23767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } }
I20260924 23:20:17.241938 23890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } } }
I20260924 23:20:17.242115 23740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:20:17.242131 23890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:20:17.243031 23740 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2 [term 1 FOLLOWER]: Refusing update from remote peer 18fe3620a8df4ceb8fd1bc0b52771618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:20:17.242115 23741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "19818426417c4b7aae19263da66c76a2"
I20260924 23:20:17.243393 23862 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:20:17.244110 23741 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18fe3620a8df4ceb8fd1bc0b52771618 in term 1.
I20260924 23:20:17.242018 23767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:20:17.245381 23890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18fe3620a8df4ceb8fd1bc0b52771618. Latest consensus state: current_term: 1 leader_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } } }
I20260924 23:20:17.245502 23890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:20:17.245576 23890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } } }
I20260924 23:20:17.245815 23890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 23:20:17.251911 23904 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:20:17.251981 23904 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:20:17.252395 23893 mvcc.cc:205] Tried to move back new op lower bound from 7333036102497259520 to 7333036102447628288. Current Snapshot: MvccSnapshot[applied={T|T < 7333036102497259520}]
I20260924 23:20:17.263964 23875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18fe3620a8df4ceb8fd1bc0b52771618" member_type: VOTER last_known_addr { host: "127.1.4.254" port: 40527 } } peers { permanent_uuid: "19818426417c4b7aae19263da66c76a2" member_type: VOTER last_known_addr { host: "127.1.4.253" port: 43617 } } peers { permanent_uuid: "137e5045c5a64e8dbcbfdc948fdc5e7a" member_type: VOTER last_known_addr { host: "127.1.4.252" port: 44085 } } }
I20260924 23:20:17.264108 23875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618 [sys.catalog]: This master's current role is: LEADER
I20260924 23:20:17.266081 23877 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:20:17.266824 23877 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:20:17.278002 23877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18fe3620a8df4ceb8fd1bc0b52771618: Generated new TSK 0
I20260924 23:20:17.278276 23877 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 23:20:17.304141 23695 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:17.302292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.1.4.254:40527)
0924 23:20:17.302339 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.302341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.303435 (+ 1094us) server_negotiation.cc:407] Connection header received
0924 23:20:17.303540 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.303541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.303553 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.303583 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.304065 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41872: BlockingRecv error: recv got EOF from 127.0.0.1:41872 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 23:20:17.305011 23771 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:17.302818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.1.4.253:43617)
0924 23:20:17.302861 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.302863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.304467 (+ 1604us) server_negotiation.cc:407] Connection header received
0924 23:20:17.304561 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.304562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.304573 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.304614 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.304940 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42642: BlockingRecv error: recv got EOF from 127.0.0.1:42642 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 23:20:17.305867 23846 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:17.303154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.1.4.252:44085)
0924 23:20:17.303213 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.303215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.305307 (+ 2092us) server_negotiation.cc:407] Connection header received
0924 23:20:17.305427 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.305429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.305440 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.305469 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.305806 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54912: BlockingRecv error: recv got EOF from 127.0.0.1:54912 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 23:20:17.308342 23771 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:17.307571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.1.4.253:43617)
0924 23:20:17.307632 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.307633 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.307841 (+ 208us) server_negotiation.cc:407] Connection header received
0924 23:20:17.307941 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.307942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.307962 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.307989 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.308294 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42646: BlockingRecv error: recv got EOF from 127.0.0.1:42646 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 23:20:17.311493 23771 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:17.310660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42650 (local address 127.1.4.253:43617)
0924 23:20:17.310693 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.310695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.310952 (+ 257us) server_negotiation.cc:407] Connection header received
0924 23:20:17.311059 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.311060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.311072 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.311100 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.311443 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42650: BlockingRecv error: recv got EOF from 127.0.0.1:42650 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 23:20:17.315438 23771 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:17.314659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.1.4.253:43617)
0924 23:20:17.314697 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.314699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.314931 (+ 232us) server_negotiation.cc:407] Connection header received
0924 23:20:17.315017 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.315018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.315028 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.315055 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.315354 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42666: BlockingRecv error: recv got EOF from 127.0.0.1:42666 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 23:20:17.321533 23771 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:17.320654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.1.4.253:43617)
0924 23:20:17.320695 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.320697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.320958 (+ 261us) server_negotiation.cc:407] Connection header received
0924 23:20:17.321070 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.321071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.321083 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.321111 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.321477 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42674: BlockingRecv error: recv got EOF from 127.0.0.1:42674 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 23:20:17.331995 23771 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:17.330980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.1.4.253:43617)
0924 23:20:17.331023 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.331025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.331336 (+ 311us) server_negotiation.cc:407] Connection header received
0924 23:20:17.331480 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.331481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.331495 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.331524 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.331934 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42684: BlockingRecv error: recv got EOF from 127.0.0.1:42684 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 23:20:17.350330 23771 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:17.349374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.1.4.253:43617)
0924 23:20:17.349415 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.349417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.349724 (+ 307us) server_negotiation.cc:407] Connection header received
0924 23:20:17.349825 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.349826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.349840 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.349870 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.350278 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42700: BlockingRecv error: recv got EOF from 127.0.0.1:42700 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 23:20:17.384382 23771 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:17.383612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.1.4.253:43617)
0924 23:20:17.383661 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.383664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.383838 (+ 174us) server_negotiation.cc:407] Connection header received
0924 23:20:17.383904 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.383905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.383923 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.383971 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.384318 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42716: BlockingRecv error: recv got EOF from 127.0.0.1:42716 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 23:20:17.450134 23771 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:17.449438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.1.4.253:43617)
0924 23:20:17.449504 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.449506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.449624 (+ 118us) server_negotiation.cc:407] Connection header received
0924 23:20:17.449676 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.449677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.449689 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.449739 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.450071 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42730: BlockingRecv error: recv got EOF from 127.0.0.1:42730 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 23:20:17.580086 23771 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:17.579315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.1.4.253:43617)
0924 23:20:17.579389 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.579391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.579482 (+ 91us) server_negotiation.cc:407] Connection header received
0924 23:20:17.579550 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.579551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.579563 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.579661 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.579991 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42736: BlockingRecv error: recv got EOF from 127.0.0.1:42736 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 23:20:17.837901 23771 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:20:17.837293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42740 (local address 127.1.4.253:43617)
0924 23:20:17.837364 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:17.837366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:17.837379 (+ 13us) server_negotiation.cc:407] Connection header received
0924 23:20:17.837449 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:17.837450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:17.837460 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:17.837510 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:20:17.837839 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42740: BlockingRecv error: recv got EOF from 127.0.0.1:42740 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 23:20:18.181849 23859 catalog_manager.cc:1259] Loaded cluster ID: 5e599c4648fc47ddac4fdb769b5e2c35
I20260924 23:20:18.181921 23859 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a: loading cluster ID for follower catalog manager: success
I20260924 23:20:18.182701 23859 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a: acquiring CA information for follower catalog manager: success
I20260924 23:20:18.182953 23859 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 137e5045c5a64e8dbcbfdc948fdc5e7a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 23:20:18.252437 23904 catalog_manager.cc:1259] Loaded cluster ID: 5e599c4648fc47ddac4fdb769b5e2c35
I20260924 23:20:18.252519 23904 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2: loading cluster ID for follower catalog manager: success
I20260924 23:20:18.253468 23904 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2: acquiring CA information for follower catalog manager: success
I20260924 23:20:18.253753 23904 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 19818426417c4b7aae19263da66c76a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 23:20:18.353487 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:18.351011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.1.4.253:43617)
0924 23:20:18.351115 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:18.351117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:18.351172 (+ 55us) server_negotiation.cc:407] Connection header received
0924 23:20:18.351202 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:18.351202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:18.351253 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:18.351325 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:20:18.351471 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:18.351985 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:18.352803 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:18.353016 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:18.353068 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:18.353208 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:20:18.353316 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:20:18.353372 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:18.353389 (+ 17us) server_negotiation.cc:299] Negotiation successful
0924 23:20:18.353405 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":53,"threads_started":1}
I20260924 23:20:18.357529 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:18.355197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.1.4.252:44085)
0924 23:20:18.355305 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:18.355307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:18.355314 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:20:18.355349 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:18.355350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:18.355418 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:18.355489 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:20:18.355588 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:18.356058 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:18.356826 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:18.357017 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:18.357061 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:18.357239 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:20:18.357342 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:20:18.357398 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:18.357428 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 23:20:18.357445 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":62,"threads_started":1}
I20260924 23:20:18.361933 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:18.359286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.1.4.252:44085)
0924 23:20:18.359324 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:18.359326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:18.359561 (+ 235us) server_negotiation.cc:407] Connection header received
0924 23:20:18.359654 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:18.359655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:18.359666 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:18.359692 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:20:18.359849 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:18.360582 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:18.361359 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:18.361549 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:18.361603 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:18.361722 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:20:18.361753 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:20:18.361795 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:18.361839 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 23:20:18.361858 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:20:18.362419 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:18.359245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42760 (local address 127.1.4.253:43617)
0924 23:20:18.359332 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:18.359334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:18.359619 (+ 285us) server_negotiation.cc:407] Connection header received
0924 23:20:18.359688 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:18.359689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:18.359701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:18.359758 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:20:18.359889 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:18.360607 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:18.361578 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:18.361847 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:18.361888 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:18.362123 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:20:18.362155 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:20:18.362184 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:18.362319 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 23:20:18.362347 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 23:20:18.362546 23983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:20:18.359247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41884 (local address 127.1.4.254:40527)
0924 23:20:18.359392 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 23:20:18.359394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:20:18.359445 (+ 51us) server_negotiation.cc:407] Connection header received
0924 23:20:18.359559 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:20:18.359560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:20:18.359619 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:20:18.359693 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:20:18.359836 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:18.360357 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:18.361388 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:20:18.361694 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:20:18.361772 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:20:18.362019 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:20:18.362133 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:20:18.362194 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 23:20:18.362428 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 23:20:18.362452 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":43,"threads_started":1}
I20260924 23:20:18.364207 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 23624
I20260924 23:20:18.368711 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 23699
I20260924 23:20:18.373394 1043 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEEvosl/build/release/bin/kudu with pid 23774
2026-09-24T23:20:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2712 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5437 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (262922 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST